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Executive  Summary 


The  Deputy  Assistant  Secretary  of  Defense  for  Developmental  Test  and  Evaluation 
[DASD(DT&E)]  tasked  IDA  to  “collect,  organize,  and  assess  best  practices  and  lessons 
learned  (BPLL)  for  performing  and  supporting  DT&E  by  government  and  industry; 
provide  tools  for  DASD(DT&E)  staff  specialists,  program  offices,  and  test  organizations 
to  provide  faster  response  in  DT&E  planning  and  execution;  and  provide  an  organized, 
easily  accessible,  and  interactive  web-based  DoD-level  repository  of  T&E  BPLL 
available  for  the  acquisition  and  T&E  communities.”1 

The  task  involved  several  distinct  subtasks.  First,  we  collected  lessons  learned  and 
best  practices  from  previous  experiences  and  derived  or  adapted  best  practices  for  general 
use.  Next,  we  developed  the  necessary  software  and  protocols  to  establish  the  DT&E 
Forum  website  on  a  Webserver  operated  by  the  Office  of  the  Under  Secretary  of  Defense 
for  Acquisition,  Technology,  and  Logistics  [OUSD(AT&L)].  Finally  we  assessed  the 
potential  for  the  DT&E  Forum  to  be  the  model  for  DT&E’s  Internal  Website.  This  paper 
describes  our  approach,  the  activities  used  to  carry  out  the  tasking,  and  the  results 
obtained. 

Collection  of  Material  and  Development  of  Best  Practices 

The  purpose  of  this  subtask  was  to  research  and  analyze  best  practices  and  lessons 
learned  materials  from  various  sources  and,  from  that  material,  to  derive  “Best  Practices” 
that  would  be  useful  in  the  planning,  execution,  evaluation,  and  reporting  of  DT&E  for 
DoD  acquisition  and  development  programs.  The  final  aspect  of  the  subtask  was  to 
fonnat  the  information  so  it  would  be  suitable  for  posting  to  and  accessing  from  the 
DT&E  Forum  website  hosted  by  OUSD(AT&L).  This  subtask  required  the  team  to: 

•  Research  and  acquire  T&E  best  practices  and  lessons  learned  from  Components, 
industry,  professional  associations,  the  Government  Accounting  Office  (GAO), 
and  the  Defense  Acquisition  University  (DAU)  Best  Practices  Clearinghouse. 

•  Interview  DT&E  staff  specialists,  test  centers,  and  representative  program 
management  offices  to  collect  experiences  on  T&E  best  practices  and  lessons 
learned,  as  well  as  draw  on  the  experiences  of  IDA  team  members. 
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IDA  Task  Order  AX-1 -3334,  “Best  Practices  and  Lessons  Learned  for  Deputy  Assistant  Secretary  of 
Defense  for  Developmental  Test  and  Evaluation.” 


The  effort  began  with  a  review  of  approximately  30  “lessons  learned”  documents 
provided  by  the  sponsor.  From  this  review,  the  team  derived  an  initial  set  of  Best 
Practices.  This  review  also  led  to  the  establishment  of  a  standardized  format  for  Best 
Practices  and  the  development  of  a  taxonomy  for  organizing  them.  The  format  included 
a  unique,  action-oriented  title;  a  bulleted  checklist  of  best  practice  actions;  the  lesson(s) 
learned  that  led  to  development  of  the  best  practice;  hyperlinks  to  sources  from  which 
material  was  drawn;  and  the  milestone  or  phase  of  the  DT&E  process  to  which  the  best 
practice  relates.  The  taxonomy  was  divided  between  topics  pertinent  to  assisting 
program  offices  and  test  program  planners  structure  their  acquisition  and  development 
programs  to  get  them  through  the  Defense  Acquisition  System  milestones;  and  those  to 
aid  testers  in  developing  and  executing  test  plans  for  commodities  in  warfare  areas  such 
as  munitions  and  armaments,  ground  vehicles,  air  platforms,  sea  vehicles,  space 
platforms,  nuclear  technology,  battlespace  environments,  and  infonnation  systems. 

The  team  followed  this  initial  effort  by  assembling  more  than  250  documents  related 
to  DT&E  lessons  learned  and  best  practices.  These  documents,  along  with  infonnation 
collected  in  interviews  and  applicable  experience  residing  in  the  team,  were  used  to  draft 
a  large  number  of  candidate  best  practices,  which  were  vetted  by  both  the  team  and  others 
at  IDA.  From  this  vetting,  the  team  selected  an  initial  set  of  76  for  sponsor  review. 
These  reviews  were  attended  by  several  levels  of  DASD(DT&E)  management  including 
the  Principal  Deputy  Director,  the  Military  Assistant,  Deputy  Directors,  and  staff 
specialists.  Of  the  76,  24  were  selected  for  final  posting  on  the  DT&E  Forum  for  beta 
testing  with  the  DT&E  community.  An  abbreviated  example  of  a  best  practice,  showing 
the  format,  is  provided  below. 

In  the  course  of  developing  the  Best  Practices,  two  overarching  findings  stood  out. 
First,  there  is  a  significant  need  for  collecting  and  making  available  additional  best 
practices  including  a  number  of  which  are  suggested  in  this  report.  Second,  from 
discussions  with  people  actually  engaged  in  DT&E  efforts  including  field  test 
organizations  and  DASD(DT&E)  staff  specialists,  “planning  factors”  emerged  as  an  area 
of  high  interest  and  needs  additional  and  continued  development  for  use  in  building 
evaluation  frameworks  for  test  program  schedules,  resource  planning,  and  costs  in  Test 
and  Evaluation  Master  Plans.  They  indicated  that  a  key  element  in  such  a  framework  is 
the  planning  factors  used  by  program  offices  to  scale  analogous  historical  test  program 
data  for  current  programs. 
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BP  11  -  015:  Establish  a  Test  and  Evaluation  Working- 
Level  Integrated  Product  Team  (T&E  WIPT) 

A.  Best  Practice 

Charter  a  T&E  WIPT  to  advise  and  develop  the  T&E  program 
for  TEMPs  and  acquisition  strategies,  cost  and  schedule  planning 


As  early  in  an  acquisition  program,  preferably  at  the  onset  of  a 
materiel  solution  decision,  the  program  office  should  charter  a  T&E 
WIPT . with,  as  a  minimum,  the  following  representative  membership: 

•  The  designated  Government  Lead  DT&E  Organization  (external 
to  and  independent  of  the  program  office) 


B.  Rationale  (Lessons  Learned) 

A  T&E  WIPT  was  chartered  to  support  the  Biometrics  Automated 

Toolset  (BAT),  field  tested  in  Kosovo . attributed  for  the  successful 

rapid  response  of  this  program  and  the  eventual  adoption  of  the  capabilities 
by  other  government  agencies. 

C.  Sources 

•  “Shaping  the  Way  Ahead:  Army  Biometrics  WIPT  Kickoff,” 
Defense  AT&L,  July- August  2011,  pp.  27-32,  by  Nicole  Daniel, 
Kevin  Trissell,  and  Richard  Hansen. 

•  . 
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Abbreviated  Example  of  a  Best  Practice  Showing  the  Structure 
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Development  of  the  DT&E  Forum  Website 

The  second  sub  task  was  to  develop  a  website  containing  the  DT&E  documents  and 
other  relevant  DT&E  information  and  to  provide  two-way  communications  between  the 
Office  of  the  DASD(DT&E)  [ODASD(DT&E)]  and  those  who  are  conducting  DT&E 
activities  in  the  field. 

Meant  to  act  as  a  worldwide  source  of  infonnation  pertinent  to  the  DT&E  effort  and 
to  be  accessed  using  a  Common  Access  Card  (CAC)  login  from  any  location  having 
Internet  connectivity,  the  DT&E  Forum  is  also  intended  to  provide  a  means  for  those 
engaged  in  DT&E  activities  to  bring  to  the  attention  of  ODASD(DT&E)  applicable 
information  which,  in  the  user’s  opinion,  should  be  added  to  that  already  in  the  DT&E 
Forum. 

The  DT&E  Forum  is  currently  running  on  an  AT&L  server  at 
https://ebiz.atl.osd.mil/DTE-Forum/.  Its  home  page  provides  users  with  direct  access  to 
the  main  functions  of  the  website  including  locating  BPLL  documents  contained  in  its 
database;  enabling  users  to  suggest  additional  BPLL  documents;  enabling  users  to  track 
the  disposition  of  their  own  input;  displaying  announcements,  blogs,  and  links  to  other 
DT&E  resources;  and  providing  a  means  to  contact  DASD(DT&E)  directly. 

Potential  for  the  DT&E  Forum  to  Be  the  Model  for  DT&E’s  Internal 
Website 

As  the  task  was  nearing  completion  the  website  development  team  was  asked  the 
degree  to  which  the  DT&E  Forum  website  could  be  expanded  to  provide  additional 
services  internal  to  the  ODASD(DT&E).  Specifically,  how  difficult  would  it  be  to  make 
the  transfonnation?  Although  the  team  did  not  have  the  resources  to  answer  this  question 
in  depth,  it  was  able  to  take  the  following  steps  to  show  the  value  of  using  the  current 
DT&E  Forum  website  as  the  basis  for  a  DT&E  internal  website  and  the  ease  with  which 
the  transformation  could  be  done. 

First,  the  BPLL  application  was  separated  from  the  website  code  and  a  DT&E 
internal  website  home  page,  an  example  of  which  is  shown  below,'  was  added.  The 
BPLL  application  was  then  connected  to  the  BPLL  Documents  button  on  the  DT&E 
internal  website  home  page. 

Next,  a  quick  review  of  ongoing  tasks  at  IDA  identified  eight  for  DT&E  in  addition 
to  development  of  the  BPLL  documents  and  website  task.  These  include  tasks  for  air 
warfare,  land  warfare,  naval  warfare,  space  and  ballistic  missile  defense,  and  cyber  and 
information  systems  as  well  as  special  projects  for  the  Principal  Deputy,  DT&E. 
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It  should  be  noted  that  the  layout  of  this  home  page  is  for  demonstration  purposes  only. 


As  a  final  step,  the  team  designed  initial  applications  to  display  data  from  a  number 
of  the  identified  ongoing  tasks.  Where  resources  permitted,  the  applications  were  added 
to  the  revised  Forum  website.  The  revised  website  was  then  placed  on  IDA’s 
developmental  server  so  that  DT&E  personnel  could  access  and  examine  it.3’4 
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control  allows  the  document  to  be  shared  via  the  website’s  library. 
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1.  Background,  Objective,  Approach,  and 

Schedule 


A.  Background  and  Objective 

DASD(DT&E)  tasked  IDA  to  create  an  organized,  easily  accessible,  interactive, 
web-based,  repository  of  best  practices  and  lessons  learned  (BPLL)  for  planning, 
executing,  resourcing,  executing,  reporting,  and  reviewing  developmental  test  and 
evaluation  (DT&E)  activities.  The  purpose  is  to  provide  DASD(DT&E)  staff  specialists, 
program  offices,  and  test  organizations  with  rapid  access  to  materials  pertinent  to  their 
tasks  to  accomplish  them  in  the  most  efficient  and  effective  manner  by  drawing  on 
experiences  of  others.  Furthennore,  it  provides  users  with  a  collaborative  structure  to 
harvest  and  share  their  best  practices  and  lessons  learned  and  to  suggest  improvements  to 
current  ones. 

As  part  of  this  effort,  the  team  collected,  organized,  and  assessed  those  BPLLs 
necessary  for  Government  and  industry  to  perform  and  support  DT&E.  The  team  also 
provided  tools  for  DASD(DT&E)  staff  specialists,  program  offices,  and  test  organizations 
to  permit  faster  response  in  DT&E  planning  and  execution. 

The  resulting  repository  for  BPLLs  is  hosted  on  a  Webserver  operated  by  the  Office 
of  the  Under  Secretary  of  Defense  (Acquisition,  Technology  &  Logistics) 
(OUSD(AT&L)  under  the  name  “DT&E  Forum”  for  which  IDA  developed  the  coding 
and  protocols  for  adapting  to  the  presentation  and  security  requirements. 

B.  Approach 

This  effort  had  four  broad  phases:  BPLL  data  collection;  analysis  and  organization 
of  the  collected  data;  development  of  best  practices  based  on  the  collected  data  along 
with  experiences  of  team  members;  and  development  of  an  interactive,  web-based,  DoD- 
level  repository  capable  of  storing  and  providing  access  to  stored  BPLL  documents. 
These  phases  are  described  in  Table  1-1. 
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Table  1-1.  Four  Phases  of  Task 


Phase 

Subtasks 

Data 

Collection 

Research  and  acquire  DASD(DT&E)  best  practices  and  lessons  learned  from 
Components,  industry,  and  professional  associations. 

Interview  DT&E  staff  specialists  and  representative  program  management  offices  to 
collect  experiences  on  DT&E  best  practices  and  lessons  learned  and  further  define 
requirements  for  how  to  present  information  so  that  it  would  be  most  useful. 

Research  relevant  and  existing  program  and  other  documentation  to  include  Test  and 
Evaluation  Master  Plans  (TEMPs),  acquisition  strategies,  requests  for  proposal,  studies, 
etc.,  to  locate  and  acquire  best  practices  and  lessons  learned  for  performing  DT&E. 

Analyze  Defense  Acquisition  University  (DAU)  Best  Practices  Clearinghouse  website  to 
identify  material  relevant  to  this  effort  including  collection  methods  of  T&E  best 
practices. 

Data  Analysis 
and 

Organization 

Provide  analysis  and  recommendations  of  T&E  best  practice  collection  methods  and 
how  DASD(DT&E)  might  leverage  existing  processes  in  a  DoD-level  tool. 

Provide  analysis  and  recommendations  of  a  user-friendly  taxonomy  of  T&E  best 
practices  and  lessons  learned,  based  on  information  from  the  above  tasks,  to  be 
established  on  a  DT&E  website. 

Taxonomy  should  be  applicable  for  preparation  of  and  use  in  oversight  of  T&E  program 
documentation  to  include  TEMPs,  acquisition  strategies,  requests  for  proposals,  etc. 

Development 
of  Best 

Practice 

Documents 

Use  the  material  collected  in  the  above  tasks  along  with  the  applicable  team 
experiences  to  develop  best  practices  documents. 

Review  the  documents  with  selected  DT&E  management. 

Enter  approved  best  practices  documents  into  the  web-based  repository. 

Development 

and 

Refinement  of 

a  DT&E  Web- 

Based 

Repository 

Design,  develop,  test,  and  demonstrate  a  prototype  web-based  BPLL  site  that  can  be 
hosted  on  the  DT&E  website.  The  BPLL  site  is  to  be  searchable  and  provide  links  to 
relevant  resources  [e.g.,  Society  of  Experimental  Test  Pilots  (SETP)  website,  Test 
Resource  Management  Center  (TRMC)  Test  Capability  Directory,  etc.].  Provide  best 
practices  and  lessons  learned  content  in  approved  taxonomy  of  task  above.  The 
prototype  shall  be  hosted  on  the  IDA  interoperability  decision  support  system  (IDSS) 
network  during  the  development  phase  with  versions  to  be  put  onto  the  DT&E  website. 

Provide  and  implement  a  feedback  tool  on  the  BPLL  site  to  include  both  a  feedback 
mechanism  for  the  site  itself  and  a  user-friendly  method  for  suggesting  new  T&E  best 
practices  and  lessons  learned. 

Continuously  modify  the  prototype  site  as  directed  by  the  DASD(DT&E)  point  of  contact 
(POC). 

At  the  direction  of  the  DASD(DT&E)  POC,  aid  DT&E  in  transforming  the  prototype  to 
enable  it  to  be  hosted  on  the  DT&E  website. 

Respond  to  internal  and  external  user  feedback  and  implement  changes  per 
DASD(DT&E)  direction. 

Schedule  and 

Deliverables 

Provide  briefings  on  document  and  website  development  as  requested. 

Provide  demonstrations  of  the  website  as  requested. 

By  September  30,  2012,  enter  approved  BPLL  documents  into  updated  prototype  BPLL 
website  and  go  “live.” 
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2.  Lessons  Learned  and  Best  Practices 


This  chapter  describes  the  steps  taken  to  derive  and  format  BPLLs  suitable  for 
posting  to  and  accessing  from  the  DT&E  Forum  website. 

A.  Collection  of  Lessons  Learned  and  Best  Practices 

We  began  the  effort  by  reviewing  approximately  30  “lessons  learned”  documents 
provided  by  DASD(DT&E),  from  which  we  derived  our  first  set  of  Best  Practices, 
standardized  a  format,  and  developed  a  taxonomy  for  organizing  them.  We  assigned 
specific  documents  to  team  members  with  the  most  appropriate  experience  or  knowledge 
to  derive  best  practices  for  use  by  DT&E  personnel  in  acquisition  program  offices,  test 
organizations,  and  staff  specialists  in  the  Office  of  the  DASD(DT&E).  This  was 
followed  by  researching  and  collecting  additional  lessons-learned  documents  from  the 
IDA  archives  and  library.  This  endeavor  produced  more  than  250  documents  from 
sources  such  as  the  Government  Accountability  Office  (GAO),  Defense  Science  Board 
(DSB),  DoD  contractors,  and  non-DoD  industry;  DoD  and  other  Government  agency 
special  studies;  professional  society  publications,  forums,  briefings,  and  podcasts  such  as 
the  Society  of  Experimental  Test  Pilots  (SETP),  International  Test  and  Evaluation 
Association  (ITEA),  American  Institute  of  Aeronautics  and  Astronautics  (AIAA),  U.S. 
Naval  Institute,  and  Association  of  U.S.  Anny;  and  commercial  publications  such  as 
Aviation  Week.  This  was  followed  by  interviews  with  DASD(DT&E)  staff  specialists, 
and  on-site  visits  to  test  organizations  such  as  the  Air  Force  Flight  Test  Center  at 
Edwards  AFB,  NAVAIR  Aircraft  Division  Patuxent  Naval  Air  Station,  46th  Test  Wing  at 
Eglin  AFB,  the  Air  Force  Operational  Test  and  Evaluation  Command  at  Kirtland  AFB, 
and  the  Weapons  System  Evaluation  Program  (WSEP)  Program  at  Tyndall  AFB.  We 
also  sent  a  representative  to  the  annual  meeting  of  the  SETP,  which  we  found  to  be  an 
excellent  source  of  lessons  learned.  We  held  meetings  with  Office  of  the  DASD(DT&E) 
deputies  and  staff  specialists  to  get  their  ideas  of  issues  and  topics  they  would  like  to  see 
covered  in  best  practices. 

B.  Development  of  Best  Practices 

Having  collected  lessons  learned  and,  in  some  cases,  suggested  best  practices  from 
experiences  of  our  IDA  team  members,  the  next  step  was  to  determine  which  would  be 
the  most  useful  in  deriving  best  practices  for  the  DT&E  community;  i.e.,  would  have  the 
greatest  payoff.  For  example,  we  asked  if  the  lessons  learned  were  likely  to  (1)  be 
beneficial  to  all,  or  at  least  applicable  to  certain  classes  of,  DT&E  programs;  (2)  result  in 
cost  savings  or  avoidances  by  improving  efficiency  and  effectiveness  of  planning, 
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performing,  evaluating,  and  reporting  DT&E  early  in  the  program  to  detect  design  and 
function  flaws  when  it  is  least  expensive  to  correct  them  (i.e.,  preproduction);  and  (3) 
save  acquisition  program  schedule  time  by  reducing  TEMP  approval  iterations  and 
testing  rerun  requirements. 

We  found  all  best  practices  generally  could  be  grouped  under  one  of  two  categories: 

•  Technical  test  procedures  for  use  by  test  or  evaluation  planners  such  as  the  Lead 
Developmental  Test  and  Evaluation  Organizations  (LDTOs),  and  Integrated 
Test  Teams  (ITTs)  organized  by  Defense  Technology  Area  Plan  (DTAP) 
warfare  area. 

•  Management/administrative  topics  for  use  by  DT&E  planners  in  program 
offices,  the  Chief  Developmental  Tester,  T&E  Working  Level  Integrated 
Product  Teams  (T&E  WIPT),  and  staff  specialists  in  Headquarters  and  OSD 
functions,  organized  according  to  Acquisition  Milestone  or  Phase,  and 
consisting  of  checklists  of  actions  that  should  be  taken  or  avoided. 

Within  the  resources  provided  under  this  task,  of  the  24  Best  Practices  accepted  for 
use  on  the  Demonstration  DT&E  Forum,  a  majority  are  for  Acquisition  Milestone/Phase. 
It  is  recommended  to  eventually  populate  the  taxonomy  to  cover  all  aspects  of  DT&E  by 
deriving  best  practices  from  one  or  more  lessons  learned  that  could  be  mapped  to  either 
procedures  for  perfonning  DT&E  in  each  of  the  major  warfare  areas,  or  for  proceeding 
through  the  Defense  Acquisition  System  milestone/phase  model.1  The  taxonomy  is 
shown  in  Figure  2-1.2. 


Figure  2-1.  Taxonomy  for  Mapping  Best  Practices  and  Lessons  Learned 


DoDI  5000.02,  Operation  of  the  Defense  Acquisition  System,  dated  December  2,  2008. 
The  DT&E  Procedure  Information  Systems  includes  communications. 
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As  an  example,  we  derived  a  best  practice  (BP  11-15)  based  on  lessons  learned 
about  the  value  of  early  involvement  of  DT&E  personnel  in  a  “Test  and  Evaluation 
Working-Level  Integrated  Product  Team  (T&E  WIPT)”  to  assist  an  acquisition  program 
office  in  developing  their  Acquisition  Strategy  during  the  “Requirements  and  Materiel 
Solutions  Analysis”  phase  of  a  new  acquisition  program.  We  envisioned  it  could  be 
mapped  or  “binned”  to  this  taxonomy  as  shown  in  Figure  2-2. 

For  both  categories,  we  developed  a  fonnat  for  presenting  the  best  practices,  which 
is  shown  in  Figure  2-3. 

As  exemplified  in  Appendix  A,  each  best  practice  has  been  assigned  a  unique  alpha¬ 
numeric  designation  representing  the  fiscal  year  in  which  it  was  initially  developed,  that 
is  “BP1 1-015”  for  2011,  and  an  action  oriented  title  “Establish  a  Test  and  Evaluation 
Working-Level  Integrated  Product  Team  (T&E  WIPT).”  The  title  is  followed  by  short 
action  statement  (in  bold)  of  the  main  thrust  of  the  derived  best  practice  followed  by 
specific  actions  (in  bullet  form)  that  can  enact  the  best  practice.  “Rationale”  contains  the 
specific  lessons  learned  from  which  the  best  practice  was  derived.  “Sources”  are  the 
names  of  the  documents  in  which  the  lessons  learned  were  found,  and  they  are 
hyperlinked  to  those  documents  for  further  detail.  “Suggested  Key  Words”  are  used  to 
aid  in  searches  for  the  topic,  with  bolded  key  words  referring  to  words  in  the  title  or  the 
best  practice  statement,  and  those  not  bolded  are  other  relevant  or  equivalent  words  used 
in  the  remainder  of  the  text  for  the  best  practice  or  in  the  rationale.  “Author,”  “Date,” 
and  “Bin  Descriptor”  may  be  hidden  on  the  DT&E  Forum  website  but  are  available  for 
use  in  follow-up  interactive  features  such  as  the  blog. 
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The  details  of  this  example  are  shown  in  Appendix  A. 

2-3 


[ 


Technology  Opportunities  &  Resources 


•  The  Materiel  Development  Decision  precedes 
entry  Into  any  phase  of  the  acquisition 
management  system 

•  Entrance  criteria  met  before  entering  phase 

•  Evolutionary  Acquisition  or  Single  Step  to 
Full  Capability 


A  A 


(Program 

Initiation) 


A 


Materiel 

Solution 

Analysis 

Technology 

Development 

Engineering  and 
Manufacturing 
Development 

Production  & 
Deployment 

Operations  & 
Support 

Materiel 

>Bsar“™ 

C 

Post-  /\Post- 
POR  A  \y  COR  A 

LRIP/IOT&E  A  Decision 

V  Review 

Pre-Systems  Acquisition/y 


Systems  Acquisition 


/\  Sustainment  / 


Decision  Point  /\=  Milestone  Review  >=  Decision  Point  If  PDR  Is  not  conducted  before  Milestone  B 


Requirements 
&  Materiel 

Milestone 

Technology 

Milestone 

EMD 

Phase 

Milestone 

Production  & 

Operations  & 

Solution 

Analysis 

Phase 

A 

Development 

Phase 

B 

c 

Deployment 

Support 

JCIDS 

Capabilities-Based  Assessment  (CBA) 
Initial  Capabilities  Document  (ICD) 
AoA  Guidance 

Capstone  Threat  Assessment 
Critical  Technology  Elements 
AoA  Development 
Evaluation  Plan 
Embedded  instrumentation 
RFPand  CDRL  DT&E  Requirements 
Government  T&E  infrastructure  Gaps 
Modeling  and  Simulation 
RTO  Designation 
Acquisition  Strategy 
Urgent  Needs 


DT &E  Role  in  Acquisition  Strategy  Requirements  (Word  Doc) 


Acquisition  Strategy  DT&E  Contents  (PDF) 


Milestone  ATEMP  Relationship  with  Acquisition  Strategy 
(Hyperlink  with  Automated  TEMP) 


Source 

Documents 

t- 


Figure  2-2.  Example  of  Mapping  a  T&E  WIPT  to  Acquisition  Strategy  Best  Practice 


Best  Practice  Number  and  Title 

The  title  should  be  short,  yet  descriptive. 

Best  Practice 

Statemen  t  of  Best  Practice 
Checklistof  Actions 

Bulletize  multiple  BPLLs  as  “checklist-like”  actions  that  should  betaken  or 
recommended  based  onorinferredfromasource(s)documentto  be 
identified  in  the  “Rationale”  action. 

Rationale  (Lessons  Learned) 

Describe  lessons  learned  from  which  Best  Practice  is  derived  including  test 
or  acquisition  program  names,  studies,  reports,  instances,  orfailures. 

Sources  (References) 

Full  titleof  sources 
Suggested  Keywords 

Use  wo  rd  s  th  at  wi  1 1  h  elp  d  ifferentiate  th  e  ap  plication  of  th  is  BPLL  fro  m 

others 

Author 

Name,  phone,  email 
Date 

Date  of  latest  versio  n 
Bin  Descriptor 

Where  this  BPLL  should  be  binned  on  the  DASD(DT&E)  website. 


Figure  2-3.  Example  of  Best  Practice  Format 

We  established  the  following  guidelines  for  authors  in  preparation  of  each  best 
practice: 

•  Short  and  concise  statements  (checklist  style)  should  be  used  when  describing: 

-  What  should  be  tested  and  how 
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-  Actions  that  should  be  taken  per  Defense  Acquisition  System  milestone  and 
phase 

•  Style 

-  Written  for  young  and  inexperienced  readers 

-  Checklist  of  best  practices 

-  One  or  two  paragraphs  of  rationale  (lessons  learned) 

-  Attribution  to  source 

-  Hyperlinked  words  or  document  titles  to: 

o  Source  and  reference  documentation 
o  More  infonnation 

•  Keywords  (used  for  fast  search) 

-  Those  that  most  accurately  describe  the  topic  should  be  bolded. 

-  Those  that  appear  somewhere  in  the  document  and  therefore  might 
indirectly  lead  a  reader  to  this  best  practice  should  not  be  bolded. 

C.  Best  Practices  Vetting  and  Posting  on  the  DT&E  Forum 

After  internal  IDA  vetting,  76  best  practices  were  considered  worthy  of  further 
vetting  by  the  DASD(DT&E)  Military  Assistant  and  the  Program  Assessment  Deputy 
Directors.  Of  these,  24  were  selected  for  final  posting  on  the  DT&E  Forum  for  beta 
testing  with  the  T&E  community.  Summaries  of  final  best  practices  and  those  that  were 
in  process  at  the  conclusion  of  the  task  are  in  Appendix  B.  A  glossary  of  acronyms  used 
in  the  best  practices  is  in  Appendix  H. 

In  providing  the  Best  Practices  for  filing  and  vetting,  we  used  what  we  called  the 
“Super  PDF”  fonnat,  which  pennitted  embedding  all  reference  materials  into  one 
document  via  hyperlinking  internally;  i.e.,  each  best  practice  is  filed  as  a  complete 
package  with  all  references.  Microsoft  Word™  format  was  used  to  post  each  best 
practice  on  the  DT&E  Forum  website  with  the  references  in  either  Word  or  PDF  format 
accessible  by  replying  positively  to  a  prompt  “View.” 

D.  Findings  and  Conclusions 

This  subtask  developed  a  number  of  best  practices  that  were  mostly  aimed  at  DT&E 
personnel  in  acquisition  program  offices,  and  some  procedures  for  flight  testing  of 
aircraft  in  the  air  warfare  areas.  Additional  best  practices  need  to  be  developed  to: 

1 .  Reflect  changes  to  the  update  to  Department  of  Defense  Instruction  (DoDI) 

5000.02,  “Operation  of  the  Defense  Acquisition  System,”  when  approved. 

2.  Outline  T&E  scope-resource-cost  “Evaluation  Framework”  that  provides 

program  managers,  Chief  Developmental  Testers,  Lead  DT&E  Organizations, 

members  of  T&E  WIPTs,  ITTs,  and  other  decision  makers  with  the  tools  and 
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worksheets  necessary  to  give  insight  into  the  adequacy  of  a  T&E  program  from 
a  cost,  schedule,  and  performance  risk  perspective. 

3.  Promulgate  best  practice  test  procedures  for  other  DTAP  warfare  areas  beyond 
air  warfare,  which  was  emphasized  in  this  effort. 

4.  Further  define  realistic  planning  factors  for  all  warfare  areas. 

The  SETP  best  practices  that  were  in  development  when  the  subtask  went  live  for 
beta  testing  should  also  be  completed. 

In  the  course  of  developing  these  best  practices,  one  area  that  emerged  to  be  of  most 
interest  and  need  to  DASD(DT&E)  staff  specialists  for  further  and  continued 
development  is  “planning  factors.”  The  lack  of  the  incorporation  of  “realistic”  planning 
factors  in  acquisition  and  DT&E  strategies,  and  in  master  schedules  and  evaluation 
frameworks  is  felt  to  be  a  major  shortfall  of  TEMPs,  and  the  initial  work  for  the  air 
warfare  area  performed  during  this  subtask  proved  to  be  highly  regarded  by  the  test 
centers  interviewed. 
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3.  Development  of  the  DT&E  Forum  Website 


The  second  part  of  the  task  was  to  develop  a  website,  hosted  on  a  Government 
server,  which  would  contain  relevant  DT&E  documents  along  with  other  infonnation. 
This  website,  referred  to  as  the  DT&E  Forum,  is  meant  to  provide  two-way 
communications  between  the  Office  of  the  Deputy  Assistant  Secretary  of  Defense  for 
DT&E  [ODASD(DT&E)],  program  offices,  and  those  who  conduct  DT&E  activities  in 
the  field.  Intended  as  a  worldwide  source  of  DT&E  BPLL  documents,  along  with  other 
applicable  infonnation  (which  can  be  accessed  using  a  CAC)  login  from  any  location 
with  Internet  connectivity),  the  DT&E  Forum  is  also  intended  to  provide  a  means  for 
those  engaged  in  DT&E  activities  to  bring  additional  BPLL  documentation  and  other 
applicable  information  to  the  attention  of  ODASD(DT&E). 

This  chapter  describes  the  evolution  of  the  website  from  the  initial  description  of  the 
requirements  to  the  publishing  of  the  final  version  currently  hosted  on  a  AT&L  eBusiness 
server.1  It  starts  with  an  overview  of  the  DT&E  Forum  website  as  it  existed  when  this 
report  was  written.  It  continues  with  a  discussion  of  the  development  of  the  requirements 
followed  by  a  review  of  the  steps  needed  to  transform  the  requirements  into  a  process 
running  on  a  Government  server. 

A.  Overview  of  DT&E  Forum  Website 

The  DT&E  Forum’s  Home  page  is  shown  in  Figure  3-1."  Here  the  user  has  direct 
access  to  all  seven  of  the  main  functions  of  the  website,  including  locating  BPLL 
documents,  suggesting  BPLL  documents,  and  tracking  one’s  input.  Each  of  these 
functions  is  briefly  described  below.  Additional  detail  is  provided  in  the  remainder  of 
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this  chapter  and  in  the  User’s  Guide. 


At  the  time  of  this  publication,  the  URL  for  the  DT&E  Forum  website  running  on  the  AT&L  server  was 
https://ebiz.acq.osd.mil/DTE-Forum/. 

When  one  first  accesses  the  DT&E  Forum,  the  website  displays  the  Terms  of  Use  page  as  required  by 
DoD.  After  reading  the  Terms  of  Use,  clicking  the  Continue  button  at  the  bottom  of  the  page  or  the 
Home  button  at  the  top  of  the  page  brings  up  the  DT&E  Forum’s  Home  page. 

A  User’s  Guide,  a  document  required  by  the  government  organization  that  hosts  the  DT&E  Forum 
website,  is  in  Appendix  D.' 
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Figure  3-1.  DT&E  Forum’s  Home  Page 


1.  Searching  for  BPLL  Documents 

The  DT&E  Forum  website  contains  DT&E  Best  Practice  and  Lessons  Learned 
(BPLL)  documents  along  with  the  sources  from  which  they  were  derived.  A 
comprehensive  search  engine  is  contained  in  the  website  to  help  users  locate  documents 
in  the  database.  The  graphic  at  the  top  center  of  the  home  page  contains  a  text  window 
and  a  Search  button,  which  together  can  be  used  to  conduct  quick  searches  for  documents 
contained  in  the  DT&E  Forum’s  database.  Accessing  BPLL  documents  and  the 
advanced  facility  are  covered  in  the  next  section. 


2.  Suggesting  BPLL  Documents 

Users  are  encouraged  to  suggest  best  practices  and  lessons  learned  for  inclusion  in 
the  DT&E  Forum.  A  form  for  entering  a  BPLL  document  can  be  obtained  by  clicking  on 
the  Suggest  Best  Practice  (or  Submit  Lesson  Learned)  button  on  the  left  side  of  the  Home 
page,  as  shown  in  Figure  3-1. 


3.  Tracking  One’s  Input 

The  My  Stuff  function  is  a  means  for  users  to  maintain  awareness  as  to  the  status  of 
their  suggestions  and  comments.  It  is  accessible  through  the  button  labeled  My  Stuff  at 
the  center  of  the  three  buttons  just  below  the  Recent  Additions  graphic.  Clicking  on  it 
takes  the  user  to  a  page  which  lists  and  permits  the  user  to  interact  with  all  inputs  the  user 
has  made. 
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4.  Announcements 

On  the  left,  toward  the  bottom,  is  a  graphic  in  which  DT&E  can  place  infonnation 
announcing  DT&E  activities,  such  as  Blog  Events.  This  graphic  also  contains  a  text 
window  and  a  Submit  button,  by  which  the  user  can  suggest  new  Blog  Topics. 

5.  Blogging 

Blogging  is  one  means  DT&E  plans  to  use  to  obtain  Forum  users’  comments  on 
topics  of  interest.  The  Blog  graphic,  on  the  right  side  of  the  Home  page,  displays  the 
Current  Blog  Topic  and  user  comments.  It  also  contains  a  Previous  Blog  Topic  button, 
which  can  be  used  to  view  archived  topics  and  user  comments. 

6.  Links  to  Other  DT&E  Resources 

On  the  right  side  of  the  top  border  is  a  button  labeled  Links.  Clicking  this  button 
takes  the  user  to  a  page  listing  URLs  of  relevant  DT&E  resources. 

7.  Contacting  DT&E 

Also  on  the  right  side  of  the  top  border  is  a  Contact  button,  which  brings  up  a 
window  so  the  user  can  email  DT&E. 

B.  Accessing  BPLL  Documents  and  the  Advanced  Search  Capability 

A  main  objective  of  the  DT&E  Forum  website  is  to  provide  ready  access  to  DT&E 
BPLL  documents.  The  means  for  accessing  documents  stored  in  the  DT&E  Forum’s 
database  are  embodied  in  the  processes  described  in  this  section:  Quick  Search, 
Advanced  Search,  and  Viewing  BPLL  Documents. 

1.  Quick  Search 

To  quickly  locate  BPLL  documents  in  the  DT&E  Forum’s  database,  users  can  enter 
one  or  more  keywords  or  phrases4  they  would  logically  expect  to  be  associated  with  the 
document(s).  Users  can  enter  this  in  the  text  box  containing  the  words  “Find  the  gouge!” 
in  the  center  of  the  Home  page  and  then  click  on  the  Search  button  to  the  right  of  the  text 
box.5  This  will  bring  up  a  search  results  page  showing  document  titles  (ordered  by  best 
matches)  stored  in  the  database  and  containing  the  entered  keyword(s)  or  phrase(s)  in  the 
document’s  title,  narrative  or  rationale.  An  example  search  results  page  is  shown  in 
Figure  3-2. 


4  Users  should  separate  multiple  words  and  phrases  with  spaces  and  place  a  single  quote  (‘)  before  and 
after  each  phrase  (for  example,  ‘Milestone  B’).  The  search  function  will  find  keywords  as  well  as 
words  that  begin  with  the  keyword.  It  will  only  find  phrases  that  exactly  match  the  entered  phrase. 

5  Note:  this  is  a  global  ‘OR’  search.  That  is,  it  will  search  for  documents  that  contain  any  of  the 
keywords/phrases  that  have  been  entered.  If  you  need  an  ‘AND’  search  -  one  that  finds  documents 
containing  all  of  the  keywords/phrases  that  have  been  entered  -  use  the  Advanced  Search  option  as 
described  below. 
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Aircraft 


BP  11-014:  Configuration  Traceability  Impacts  on  Reporting  Test  Results 

BP  11  -  007:  Perform  Ground  Developmental  Test  and  Evaluation  (DT&E)  for  Aircraft  Stores 

Compatibility  Prior  to  Flight  Test 

BP  11  -  061:  Knowledge-based  Developmental  Test  and  Evaluation  (DT&E)  of  Product 
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BP  11  -0fi7:  Applicability  of  Commercial  Industry  Rest  Practices  in  TRF  tn  DnD  Development 


Figure  3-2.  Search  Results  Page 


The  Search  Results  page  lists  the  title  of  each  document  found  that  satisfied  the 
search  criteria.  Following  the  best  practices,  the  Results  page  lists  lessons  learned 
documents,  blog  topics,  and  links  found  in  the  search.  Also,  under  the  words  Search 
Results,  the  keywords(s)  and  phrases(s)  used  in  the  search  are  given.  In  the  example  in 
Figure  3-2,  the  keyword  was  “Test.” 

Clicking  on  the  page  icon  to  the  left  of  a  document  will  open  the  document  for  the 
user  to  read.  This  is  described  in  the  section  below  titled  “Viewing  BPLL  Documents.” 


2.  Advanced  Search 

The  website  also  contains  an  advanced  search  facility,6  shown  in  Figure  3-3.  At  the 
top,  one  can  select  the  specific  type(s)  of  material  for  which  to  search  and  the  range  of 
dates  to  be  used  in  the  search.  In  the  middle  of  the  page,  users  can  select  bins  and 
topics.7  Bins  contain  information  that  is  augmented  by  material  provided  when  a 
document  is  entered  into  the  website.  If  the  list  does  not  contain  all  the  bin  topics  a  user 


The  advanced  search  facility  is  accessible  via  the  button  directly  below  the  text  window  used  to  enter 
words  and  phrases  to  be  searched  for. 

Clicking  on  a  bin  (blue  bar  with  white  letters)  will  bring  up  a  dropdown  window  showing  topics 
associated  with  that  bin. 
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wishes  to  use  in  a  search,  up  to  three  additional  keywords/phrases  can  be  entered  at  the 
bottom  of  the  form.  Clicking  on  either  Search  button  will  initiate  the  search. 


Sources  to  Starch:  E  Eest  P*»ctces  E  lessons  -eo-ned  E&ogs  □  Links 

Dates  to  Search:  0  Seanoi  ah  Dews:  OR  fw" : 
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Coa  I ic  fee  i  ii  me.  to 


Technology  Development 
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Production  &  Development 


Operations  &  Support 


Enter  up  to  3  keywords  or  phre«  below  (Match  ■>  Any  Keynord  phrase  beloir  All  Keywords,  phrases  belotr 


About  Disclaimer  Terms  of  Use  Privacy  Admin 


Figure  3-3.  Advanced  Search  Facility 


Search  results  will  be  displayed  in  the  same  type  of  webpage  as  shown  in  Figure  3-2 
with  the  following  augmentations.  The  “Results  for:”  line  shows  the  words  searched  for 
with  “WITH”  separating  keywords/phrases  from  bin  topics.  ’  Furthennore,  both  the  bin 
and  keyword  parts  of  the  search  can  be  directed  to  operate  differently  in  that  they  both 
have  their  own  AND  and  OR  controls. 


The  ‘WITH’  separator  should  be  read  as  an  AND  operation  because  the  search  engine  will  only  locate 
information  that  meets  both  the  bin/topic  criteria  and  the  keyword  criteria. 

When  the  line  leads  off  with  ALL,  it  means  that  no  keywords  or  phrases  were  entered  in  the  text  boxes  at 
the  bottom  of  the  form. 
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3.  Viewing  BPLL  Documents 

Once  a  document  has  been  located,  as  in  Figure  3-2,  clicking  on  the  page  icon  to  the 
left  of  the  document  or  on  the  document’s  title  will  open  that  document  for  the  user  to 
read.  An  example  is  shown  in  Figure  3-4.  Note,  when  found,  the  keyword(s)  or 
phrases(s)  used  in  the  search  are  highlighted  in  the  document.  “Test”  was  used  as  the 
search  criteria  in  the  example  shown  in  the  figure. 


Figure  3-4.  Viewing  a  Document  Online 


Hyperlinks  to  documents  or  other  material  may  appear.  These  hyperlinks  are  shown 
in  blue  underlined  text  in  the  Links  to  Source  column  of  the  References  section  of  the 
document.  Clicking  on  a  hyperlink  to  a  reference  document  will  open  that  document  on 
the  screen,  in  a  separate  window,  for  the  user  to  view. 

The  top  border  of  the  document  page  contains  six  buttons  to  enable  the  user  to  carry 
out  or  request  one  of  the  following  actions:  Provide  feedback  (Comment  button),  Print, 
Download,  Highlight  Off,  Like,  and  Return  to  Results.  Highlight  Off  turns  off  the 
highlighting  of  keywords/phrases  found  in  the  search.  The  Like  button  is  available  for 
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the  user  to  indicate  that  he  or  she  likes  the  document  and  increases  the  document’s  Like 
counter  by  one.  The  Return  to  Results  button  returns  the  user  to  the  Search  Results  page. 

Clicking  on  the  Comment  button  brings  up  a  window  that  allows  users  to  enter 
comments.  If  users  do  not  want  their  names  associated  with  the  comment(s),  they  can  put 
a  check  mark  next  to  “Mark  if  not  for  Attribution.”  When  done,  the  Submit  button  sends 
the  comment(s)  to  DASD(DT&E).  With  the  MyStuff  button,  described  above,  a  user  can 
follow  actions  taken  by  DT&E  based  on  his  or  her  input. 

C.  Evolution  of  the  Website’s  Requirements 

When  the  task  began  in  May  of  201 1  no  definitive  requirements  were  available  with 
which  to  begin  to  build  the  DT&E  Forum  website.  A  number  of  meetings  were  held, 
ideas  were  presented  from  both  sides,  and  Power  Point  versions  of  websites  were 
provided.  The  home  pages  for  some  of  these  versions  are  provided  in  Figure  3-5,  3-6, 
and  3-7.  However,  it  was  not  until  DASD(DT&E)  provided  its  19  September  2011 
version10  that  sufficient  detail  and  stability  were  available  such  that  real  work  on  the 
DT&E  Forum  website  could  begin. 


View  All 


7TmE  Forum 
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Figure  3-5.  4  May  2011 


10  The  complete  set  of  slides  for  this  version  is  provided  in  Appendix  D. 
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Figure  3-6.  21  July  2011 
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Even  though  the  slides  in  the  9/19/11  PowerPoint  presentation  bear  little 
resemblance  to  the  webpages  in  the  final  Forum  website,  which  can  be  seen,  in  part,  by 
comparing  the  home  pages  in  Figure  3-8  and  Figure  3-1,  it  did,  along  with  the 
accompanying  discussion,  begin  to  make  clear  DT&E’s  main  objectives  of  the  website 
which  include  providing: 

•  Controlled  access  to  all  aspects  of  the  website  using  the  CAC  as  the  control 
element 

•  Storage  for  relevant  DT&E  best  practices  and  lessons  learned  documents, 
including  those  developed  in  the  first  part  of  this  task 

•  A  comprehensive  means  of  searching  for  and  displaying  documents  having 
topics  of  interest 

•  A  means  for  users  to  comment  on  best  practices  and  lessons  learned  contained 
in  the  database 

•  A  means  of  downloading  a  copy  of  a  selected  best  practice  or  lesson  learned 

•  A  means  for  users  to  suggest  additional  best  practice  and  lessons  learned 
documents 

•  A  means  for  users  to  comment,  directly  to  DT&E,  on  other  aspects  of  the 
website 

•  A  means  for  DT&E  to  obtain  user  reaction  to  specific  topics  via  blogs 

•  A  means  for  DT&E  to  announce  events  to  the  users 

•  Links  to  DT&E  relevant  resources. 
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Figure  3-8.  19  September  2011 
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Although  the  above  is  certainly  an  impressive  list,  some  fundamental  functions  are 
still  missing,  including  a  means  for  users  to  maintain  awareness  as  to  the  status  of  their 
suggestions  and  comments  and  the  entire  set  of  administrative  functions.  These  were 
added  by  the  programming  team.  Additions  include  the  following: 

•  User  access  to  DT&E’s  processing  of  user’s  inputs  including: 

-  Comments  on  BPLLs  contained  in  the  DT&E  Forum 

-  Suggested  additional  BPLL  documents 

-  Site  feedback 

-  Links  to  websites 

-  Blog  topics 

•  Facilities  for  Administrators  to: 

-  Manage  users 

-  Draft  new  BPLL  documents 

-  Review  and  edit  active  as  well  as  draft  BPLL  documents 

-  Manage  application  settings 

-  View  and  manage  user  submissions 

-  View  and  manage  blog  topics  and  comments 

-  Manage  DT&E  Forum  website  links  page 

-  Manage  document  categories  and  subcategories. 

D.  Timeline  for  the  Development  of  the  DT&E  Forum  Website 

About  the  time  DT&E  provided  the  9/19/11  guidance  noted  above,  they  asked  IDA 
to  provide  a  Plan  of  Action  and  Milestones  (POA&M)  for  carrying  out  and  completing 
the  effort.  IDA  provided  the  chart  shown  in  Figure  3-9.  The  3  months  allotted  for  user 
testing  was  insufficient  in  DT&E’s  mind  and  they  directed  IDA  to  revise  the  POA&M  to 
provide  7  months  of  user  testing.  The  revised  chart,  delivered  to  DT&E  on  11  October 
2011,  is  shown  in  Figure  3-10.  Subsequently,  the  target  date  DT&E  selected  for  IDA  to 
deliver  the  user  evaluation  version  of  the  website  to  AT&L  was  19  January  2012. 
However,  as  that  date  approached,  DT&E  decided  to  continue  to  modify  the  website  and 
canceled  delivery  to  AT&L.  The  resultant  schedule  is  shown  in  Figure  3-11.  As  this 
figure  shows,  the  user  evaluation  version  was  delivered  to  AT&L  on  24  May  2012.  User 
testing  was  done  during  the  period  22  June  2012  to  1  September  2012.  The  final  version 
of  the  DT&E  Forum  website  code  was  delivered  to  AT&L  about  5  September  2012  and 
AT&L  brought  the  system  operational  on  one  of  their  servers  on  27  September  2012. 
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POA&M  for  BPLL  Website  Development 
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Figure  3-9.  POA&M  for  BPLL  Website  Development 
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Figure  3-10.  Revised  POA&M  for  BPLL  Website  Development 
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Actual  POA&M  for  BPLL  Website  Development 
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Figure  3-11.  Actual  POA&M  for  BPLL  Website  Development 


E.  A  Host  for  the  DT&E  Forum  Website 

According  to  the  task  order,  IDA  was  to  deliver  the  prototype  code  and  “aid 
DASD(DT&E)  in  transfonning  the  prototype  to  enable  it  to  be  hosted  on  the  DT&E 
website.”  However,  that  turned  out  to  be  impossible  for  two  reasons.  First,  the  DT&E 
website  is  a  public  website  and,  as  such,  cannot  contain  For  Official  Use  Only 
information,  which  at  least  some  of  the  BPLL  documents  include.  Second,  the  DT&E 
website’s  pages  are  static.  They  cannot  have  associated  databases  that  are  required  to 
store  the  BPLL  documents  in  the  DT&E  Forum’s  website. 

Once  the  ODASD(DT&E)  discovered  that  they  could  not  have  the  DT&E  Forum 
website  hosted  on  their  public  website,  they  requested  help  from  AT&L.  A  meeting  to 
explore  AT&L’s  requirements  for  hosting  a  website  such  as  the  DT&E  Forum  was  held 
on  13  September  2011.  Questions  led  to  a  second  meeting  on  5  October  2011.  As  a 
result  of  these  meetings,  it  was  decided  that  the  effort  would  move  forward  with  AT&L 
hosting  the  website  on  one  of  its  servers.  AT&L’s  eBusiness  Center  provided  the  go 
forward  plan  as  an  organized  set  of  steps  (shown  in  Appendix  E).  Included  in  the  steps 
are  a  set  of  documents  AT&L  required  of  DT&E  as  part  of  the  conditions  for  AT&L  to 
host  the  DT&E  Forum  website.  These  include  but  are  not  limited  to: 
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1 .  Business  Needs  Analyses  (BNA) 

2.  Memorandum  of  Agreement  (MO  A) 

3.  Request  For  Change  (RFC) 

4.  Application  Operational  Requirements  Questionnaire 

5.  DoD  IT  Portfolio  Repository  (DITPR) 

6.  Privacy  Impact  Assessment  Form  (DD  Form  2930). 

Given  DASD(DT&E)’s  shortage  of  website  personnel  at  the  time,  IDA  was  asked  to 
fill  out  the  forms  and  send  them  to  DASD(DT&E)  for  review,  signature,  and  subsequent 
transmittal  to  AT&L.  The  completed  forms  as  transmitted  to  AT&L  are  provided  in 
Appendix  F. 

F.  Working  With  AT&L 

Some  of  the  steps  in  AT&L’s  timeline  for  deploying  the  DT&E  Forum  website  (as 
shown  in  Appendix  E)  required  a  code-level  interface  between  IDA,  ODASD(DT&E), 
and  AT&L.  These  steps  are  discussed  below. 

•  Step  10  -  IDA  delivered  all  code  and  database  and  installation/configuration 
instructions  to  AT&L  engineers.  This  and  the  next  two  steps  were  done  in 
stages,  starting  rather  early  in  the  code-development  process  and  repeated 
several  times  as  the  code  matured.  This  allowed  the  IDA  and  AT&L  teams  to 
become  acquainted  and  understand  the  testing  that  they  needed  to  do. 

•  Step  1 1  -  AT&L  ran  Fortify  (code  vulnerability)  scans  on  the  source  code  and 
sent  the  results  to  DT&E/IDA  for  review,  justification,  or  remediation. 

•  Step  12  -  IDA  modified  the  application  code  to  remediate  each  vulnerability  or 
provided  a  justification  as  to  why  a  vulnerability  did  not  need  remediation.  IDA 
provided  justifications  (in  the  fonn  of  an  Excel  spreadsheet)  and  remediated 
code  to  AT&L  for  their  review. 

•  Step  14  -  If  no  code  changes  were  required  as  a  result  of  vulnerability 
remediation,  when  documentation  was  approved  by  AT&L  IA  (Information 
Assurance),  AT&L  installed  the  approved  application  code  and  databases  into 
the  ATL  MAE  STAGING  environment.  If  changes  were  required  to 
application,  IDA  provided  updated  code  for  another  Fortify  scan  as  described  in 
steps  11  and  12. 

•  Step  15  -  Enterprise  IT  Services  Directorate  (EITSD)  conducted  a  scan  of 
STAGING  servers  with  the  application  code  loaded. 

•  Step  16  -  Installations  and  Environment  (I&E)  conducted  DT&E  User 
Acceptance  Testing  in  the  STAGING  environment. 
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•  Step  17  -  The  User  Acceptance  Testing  (UAT)  Fonn  was  signed  by 
DASD(DT&E)  and  returned  to  AT&L. 

•  After  AT&L  acceptance  of  UAT  fonn,  the  code  was  scheduled  to  be  moved  to 
the  Production  environment  during  an  after-hours  period  where  DT&E  and  IDA 
could  immediately  perform  UAT  again.  If  UAT  was  successful,  the  code 
remained  on  the  production  server.  If  UAT  failed,  the  code  was  removed  from 
the  production  server  and  the  process  of  recoding  and  testing  would  be  restarted. 
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4.  Potential  for  the  DT&E  Forum  to  Be  the 
Model  for  DASD(DT&E)’s  Internal  Website 


As  the  task  was  nearing  completion,  the  IDA  team  was  made  aware  that  a  number  of 
other  IDA  tasks  were  developing  information  requested  by  DASD(DT&E)  .  The  fonn  of 
the  information  ranged  from  groups  of  documents,  to  charts,  to  source  documents,  to  a 
software  program.  An  observation  was  made  that  at  least  some  of  this  material  would  be 
of  interest  to  a  rather  wide  DT&E  audience  and  there  did  not  appear  to  be  an  adequate 
means  of  providing  the  material  to  such  an  audience.  This  prompted  several  questions: 
“To  what  degree  can  the  DT&E  Forum  website  be  expanded  to  provide  the  needed 
distribution?”  “To  what  degree  can  the  DT&E  Forum  website  restrict  portions  of 
infonnation  and  functions  to  outside  users  to  meet  the  needs  for  a  DASD(DT&E)  Internal 
website?”  and  “What  effort  is  required  to  make  any  needed  changes?”  A  different  way  of 
looking  at  the  questions  is  to  observe  that  the  current  DT&E  Forum  website  provides 
wide-area  access  to  an  application  used  by  a  group  of  people  and  then  to  ask  about  the 
effort  required  to  modify  the  website  to  provide  many  applications  to  many  groups  of 
people.  This  chapter  represents  an  initial  attempt  to  answer  these  questions. 


A.  Utility  of  a  DASD(DT&E)  Internal  Website 

We  first  looked  at  what  use  DASD(DT&E)  might  make  of  an  internal  website. 
Some  of  the  uses  are  shown  in  the  following  list.  Although  all  of  these  functions  can  be 
done  without  an  internal  website,  the  use  of  a  website  provides  a  greater  span  of  the 
enterprise  for  both  dissemination  and  acquisition  of  information. 


Planning  of  activities 
Coordination  of  activities 
Reporting  on  activities 


Communications  among 
elements 

Collection  of  information 
Dissemination  of  infonnation. 


Next  we  asked  what  basic  functions  would  be  necessary  in  an  internal  website  and 
how  does  this  compare  with  those  in  the  DT&E  Forum  website?  Given  the  team’s  lack 
of  information  on  the  internal  website,  it  was  not  possible  to  provide  a  comprehensive 
answer  to  the  first  part  of  the  question.  However  one  can  look  at  the  basic  functions 
contained  in  the  DT&E  Forum  and  envision  how  they  might  change  in  an  internal 
DASD(DT&E)  website.  This  is  shown  in  Tables  4-1  and  4-2  for  the  user  interface  (UI) 
and  administrative  functions,  respectively.  As  can  be  seen  from  these  tables,  the  basic 
functions  in  the  DT&E  Forum  would  still  apply,  but  they  would  be  broadened.  For 
example,  from  “BPLL  documents”  to  simply  “documents.” 
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Table  4-1.  Ul  in  DT&E  Forum,  Ul  for  Internal  Website 


1.  Simple  &  complex  searches  for  BPLL  documents 

2.  Ability  to  read  and  download  BPLL  documents 

3.  Capability  to  suggest  BPs  and  LL  documentation 

4.  Blogs  for  users  to  comment  on 

5.  URLs  for  DT&E  related  websites 

6.  Direct  DT&E  contact  capability 

7.  Means  to  view  DT&E  actions  on  suggestions 


1.  Simple  &  complex  searches  for  documents 

2.  Ability  to  read  and  download  documents 

3.  Capability  to  suggest  new  documentation 

4.  Blogs  for  users  to  comment  on 

5.  URLs  for  DT&E  related  websites 

6.  Direct  DT&E  contact  capability 

7.  Means  to  view  DT&E  actions  on  suggestions 


Table  4-2.  Admin  in  DT&E  Forum,  Admin  for  Internal  Website 


1.  Manage  Users 

1.  Manage  Users 

2.  Draft  New  Best  Practices  &  Lessons  Learned 

2.  Draft  New  Documents 

3.  Review  &  Edit  Existing  BPLL  Documents 

3.  Review  &  Edit  Existing  Documents 

4.  Manage  Application  Settings 

4.  Manage  Application  Settings 

5.  Manage  User  Submission  Queue 

5.  Manage  User  Submission  Queue 

6.  Manage  Blog  Topics  and  Comments 

6.  Manage  Blog  Topics  and  Comments 

7.  Manage  Links  Page 

7.  Manage  Links  Page 

8.  Manage  Document  Categories  &  Subcategories 

8.  Manage  Document  Categories  &  Subcategories 

B.  Ongoing  DASD(DT&E)  Tasks 

at 

IDA 

A  quick  review  of  ongoing  DASD(DT&E)  tasks  at  IDA  identified  eight  areas  of 
tasking  in  addition  to  the  development  of  the  BPLL  Documents  and  Website  task.  They 
are  shown  in  Table  4-3.  All,  or  nearly  all,  are  providing  information  that  could  be  put  on 
a  website.  Some,  such  as  Air  Warfare,  are  providing  hundreds  of  pages  of  data.  Others, 
such  as  the  Knowledgebase  and  Vehicle  Reliability  tasks,  are  developing  material  meant 
for  wide  distribution.  In  the  following,  we  describe  how  selected  portions  of  this  material 
might  be  made  available  on  a  website. 


Table  4-3.  Example  of  IDA  Tasks  Creating  Documentation  for  DASD(DT&E) 


DT&E  Tasking  Area 

Examples  of  Tasking 

Air  Warfare 

Planning  Factors  -  Estimating  Test  Requirements  from  Historical  Data 

Land  Warfare 

Analyses  of  the  Reliability  and  Effectiveness  of  EO/IR  Sensors  on 
Unmanned  Aerial  Vehicles 

Naval  Warfare 

Analyses  of  Aviation  Operations  for  CVN  78 

Space  and  BMD  Systems 

Analyses  Framework  for  Mobile  User  Objective  System 

Cyber  and  Information 

Systems  (Mobile  App  Testing) 

Commercial  Practices  for  Test  and  Evaluation  of  Mobile  Applications 

Special  Projects3 

Cyber  Defense 

DT&E  of  Cyber  Defense  Capabilities 

Knowledgebase 

Development  of  DT&E  Knowledgebase 

Vehicle  Reliability 

Reliability  Quick  Reference  Guide  and  Computer  Program  for  DT&E 

These  tasks  were  done  for  the  Principal  Deputy,  DT&E. 
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C.  Potential  Forum  Website  Transformation 


To  show  how  easily  the  DT&E  Forum  website  could  be  modified  to  become  the 
DASD(DT&E)  internal  website,1  we  took  five  simple  steps.  First,  the  original  Forum 
website  was  separated  into  its  two  components:  the  generic  website  administration 
component  and  the  BPLL  application.  Second,  a  new  home  page  was  attached  to  the 
generic  website  administration  component.  As  shown  in  Figure  4-1,  this  home  page  has 
hot  buttons  for  a  number  of  applications  including  those  associated  with  land  warfare,  air 
warfare,  etc."  Third,  the  BPLL  application  was  attached  to  the  hot  button  marked  BPLL 
Documents  so  that,  when  the  button  on  the  application  is  clicked,  it  executes  as  it  did  in 
the  original  Forum.  Fourth,  we  designed  initial  applications  to  display  data  from  a 
number  of  the  ongoing  tasks.  Where  time  and  resources  permitted,  these  were  added  to 
the  proposed  internal  website.  Finally,  the  proposed  internal  website  was  placed  on 
IDA’s  developmental  server  so  that  DASD(DT&E)  personnel  could  access  and  test  it. 
The  material  available  on  the  IDA  server  at  the  time  this  document  was  written  is 
described  in  the  following  sections.  The  buttons  on  the  home  page  (Figure  4-1)  with 
applications  associated  with  them  are  underlined  with  a  solid  line.  Where  the  design  is 
described  in  the  following  sections  but  the  application  was  not  incorporated  into  the 
website,  the  underlining  is  dashed. 


Figure  4-1.  Example  of  an  Integrated  DASD(DT&E)  Forum  Home  Page 


In  the  following,  the  proposed  DT&E  internal  website  is  referred  to  simply  as  the  proposed  internal 
website. 

It  should  be  specifically  noted  that  the  layout  of  this  home  page  is  for  demonstration  purposes  only. 

At  the  time  of  this  publication  the  URL  for  the  proposed  internal  website  was 
https://www6.idss.ida.org/DTEforum2/lhome.aspx 
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1.  Air  Warfare 

The  home  page  for  the  Air  Warfare  application  is  shown  in  Figure  4-2.  The  Air 
Warfare  task  provided  information  on  planning  factors.  These  can  be  seen  by  clicking  on 
the  Air  Warfare  button  on  the  proposed  internal  website’s  home  page,  which  will  bring 
up  the  Air  Warfare’s  home  page.  Entering  any  keyword  or  phrase  contained  in  the 
template  into  the  “Find  the  gouge!”  text  box  and  clicking  on  the  Search  button  will  bring 
up  a  template  like  the  one  shown  in  Table  4-4. 


Air  Warfare  Test  Summaries 


Search  for  Summaries 

search 

.  View  Reference  Documents 

.  Add  a  Summary 
.  Admin  Functions 


Figure  4-2.  Home  Page  for  the  Air  Warfare  Test  Summaries 


Table  4-4.  Example  of  a  Planning  Factor  Template 


Heading 

Content/  Notional  Examples 

Group 

Large  Aircraft,  Air  Weapons,  Helicopter  or  Ground  Vehicle 

System  Tested 

System  Name 

Phase  of  Sys  Development 

e.g.,  DT  prior  to  Milestone  C 

Type  of  Test 

e.g..  Flight  Test 

What  is  Tested? 

(expanded) 

Expanded  description-  e.g..  Functions  covered  in  a  flight  test 

Purpose  of  Test 

e.g..  Explore  key  design  drivers;  Evaluate  program  KPP /  KSA;Test 
Program  Reqts;  Run  &  Observe;  Certification 

Venue  for  Test 

e.g.,  Edwards  Flight  Test  Center 

Test  Planning  Parameters 

Factors  and  dimensions  /Values  for  this  test  (e.g.,  days/14  days) 

Number  of  Test  Platforms 

e.g..  Three  aircraft 

Dates  of  Testing 

e.g.,  May-June  2010 

Factors  Affecting  Test 

e.g..  Range  instrumentation  problems  delayed  testing  for  1 
week. 

Did  test  Meet  Objectives? 

Short  summary  (test  quality/  system  performance,  as  relevant) 

Testing  Organization 

e.g.,  437th  Airlift  Wing 

Reference  Test  Documents 

e.g.,  XX  DT  Report,  Corp  Author,  Date,  Classification,  Pages 
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2.  Mobile  App  Testing 

Another  area  that  might  profit  from  broad  availability  of  information  is  the 
development  of  plans  for  testing  of  mobile  apps.  Figures  4-3— 4-6  show  examples  of 
some  webpages  that  could  be  used  to  help  automate  the  development  and  distribution  of 
mobile  app  test  plans.  The  first  webpage,  shown  in  Figure  4-3,  provides  the  capability 
for  the  user  to  create  a  new  checklist  and  to  view  or  modify  existing  checklists.  If  the 
user  has  the  appropriate  pennissions  and  clicks  on  the  Create  New  Checklist  button,  the 
system  will  bring  up  a  series  of  webpages,  the  first  of  which  is  shown  in  Figure  4-4. 
These  pages,  developed  using  material  from  the  Commercial  Practices  for  Test  and 
Evaluation  of  Mobile  Applications  task,  contain  a  comprehensive  list  of  issues  related  to 
the  testing  of  mobile  apps.  Not  all  issues  relate  to  every  app.  The  issues  that  need  to  be 
included  in  any  particular  test  are  determined  by  the  app’s  specification  as  well  as  how 
far  along  the  app  is  in  its  developmental  process.  The  check  list  shown  in  Figure  4-4 
contains  check  boxes  to  specify  that  a  particular  issue  is  required  (R)  to  be  included  in  the 
upcoming  test,  it  is  desired  that  it  be  included  with  a  range  of  5  to  1 ,  or  is  not  required  (N) 
to  be  included. 


App  Test  Planning  Checklist 

•  Create  New  Checklist 

•  View/Modify  Existing  Checklist  in: 

-  Air  warfare 

-  Land  Warfare 

-  Naval  Warfare 

-  Space  &  BMD 

-  Information  Systems 

•  Help 

Figure  4-3.  App  Test  Planning  Checklist 
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Creating  New  Checklist 

Issues  Include  in  Upcoming  Test  | 

APP  DOCUMENTATION 

t 

1  Is  documentation  sufficient  to  understand  Intent,  function  and  flow  of  App  such  that  ,s,nn 

an  adequate  test  plan,  which  will  answer  the  applicable  questions  in  this  document, 
can  be  created? 


LOAD,  INSTAU,  LAUNCH  AND  UNINSTALL 

1  Can  App  be  adequately  loaded  and  installed  OTA  and  via  USB  port? 

2  is  length  of  laundi/start  time  within  specified  timeframe? 

3  Can  installation  of  App  be  interrupted  and  then  App  re-installed? 

4  Can  App  be  uninstalled? 

5  Is  App  deactivatable  and  then  reactivatable? 

6  Does  App  execute  without  requinng  OS  restart? 

7  is  App  updatable  OTA  and  via  USB  port? 

8  Has  App  been  adequately  tested  on  actual  target  devices? 

9  Does  App  run  on  all  applicable  models? 


LOCALIZATION 

1  Ooes  App  correctly  handle  time  zone  changes? 

2  Does  App  permit  selection  of  target  language? 

Save  Cancel 


Figure  4-4.  Creating  New  Checklist 


At  any  point,  the  user  can  save  or  discard  (cancel)  the  new  checklist.  If  the  Save 
button  is  clicked  on,  the  system  will  bring  up  the  page  shown  in  Figure  4-5.  This 
webpage  is  used  to  record  information  about  the  checklist  as  well  as  define  where  it  is  to 
be  stored.  Once  the  organization  the  app  belongs  to  is  identified,4  a  list  of  folders 
available  for  storing  their  apps  will  be  shown.  The  user  can  select  a  folder  from  that  list 
or  define  a  new  one.  The  new  one  is  automatically  associated  with  the  selected 
organization. 


Only  organizations  for  which  the  user  has  permission  to  create  checklists  will  be  selectable. 
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4 


Save  New  Checklist 


Test  Planner's  Name:  Auto  Filled  In 

Date:  Auto  Filled  In 

App  to  be  Tested: _ 

Version: _ 

Origination  Conduction  Test: _ 

Date(s)  of  Testing: _ 


Select  Division 


T 


Select  Folder 


New 


T 


Save 


Cancel 


Figure  4-5.  Save  New  Checklist 


If  a  user  enters  the  system  to  view  or  modify  a  checklist,  he  or  she  will  see  a 
webpage  like  that  shown  in  Figure  4-3.  However,  only  those  areas  for  which  he  or  she 
has  view  or  modify  privileges  will  be  selectable.  Selecting  an  available  area  will  bring  up 
the  webpage  similar  to  that  shown  in  Figure  4-6  for  the  area  selected.  Clicking  on  the 
appropriate  button  will  cause  the  system  to  bring  up  the  checklist  for  that  test,5  which  the 
user  can  modify,  given  he  or  she  has  the  required  permissions. 


View/Modify  Existing  Checklists  In 
Air  Warfare 

1  App  1 
1  App  2 
1  App  3 

Figure  4-6.  View/Modify  Existing  Checklist  in  Air  Warfare 

3.  Space  and  Ballistic  Missile  Defense  (BMD) 

The  Mobile  User  Objective  System  (MUOS)  task  provides  DASD(DT&E)  with  a 
number  of  charts  as  part  of  an  analyses  framework  for  that  system.  Figure  4-7  shows  the 


What  is  called  a  test  here  may  actually  be  a  hierarchy  of  tests  or  test  plans  and  could  also  include  test 
results. 
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home  page  which  provides  access  to  this  information.  The  home  page  is  accessible  via 
the  Space  &  BMD  button  on  the  proposed  internal  website’s  home  page. 


Space  and  BMD 


Table/Grapic  to  Display 

•  Key  Performance  Parameters 

•  Critical  Technical  Parameters 

•  Assessment  Performance  Measures 

•  Program  Progression 

Calendar  of  Activities 
Documents 


Help 

About  Dwdwmft  Term*  of  the  Privacy  Your  Info  Admin 


Figure  4-7.  BMD  Home  Page 

Four  of  the  charts  have  been  placed  on  the  developmental  server  corresponding  to 
the  four  buttons  shown  on  the  apps  home  page:  Key  Performance  Parameters;  Critical 
Technical  Parameters;  Assessment  Performance  Measures;  and  Program  Progression. 
Clicking  on  the  Help  button  will  bring  up  a  chart,  which  provides  information  on  the 
meaning  of  the  colors  and  words  used  in  the  charts. 

4.  Knowledgebase 

The  knowledgebase  effort  collects  documents  and  other  information  of  interest  to 
DT&E  and  places  those  documents  and  information  on  a  DVD.  The  DVD  also  contains  a 
search  program  that  permits  a  user  to  search  the  contents  of  the  DVD  for  infonnation 
currently  of  interest  to  the  user.  Since  the  DT&E  Forum  was  designed  as  a  document 
repository  and  has  a  comprehensive  search  program,  it  should  be  able  to  handle  the 
knowledgebase  task  with  minor  changes  -  mostly  deletions. 

5.  Vehicle  Reliability 

The  Vehicle  Reliability  effort  has  two  types  of  material  to  place  on  the  proposed 
internal  website:  the  Reliability  Quick  Reference  Guide  and  the  software  program  that  the 
task  order  calls  for  which  is  currently  in  development. 
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6.  Library 

The  website,  in  addition  to  providing  a  library  for  each  app  or  group  of  apps,  can 
also  provide  a  ODASD(DT&E)  library.  When  a  document  is  entered  into  an  app’s 
library,  a  checkbox  can  be  provided.  The  function  of  the  checkbox  would  be  to  permit 
the  user,  as  he  or  she  is  entering  the  document,  to  indicate  that  the  document  is  to  be 
made  available  to  all  of  ODASD(DT&E)  or  only  to  the  project  team.  Documents  that  are 
made  available  to  all  of  DT&E  could  be  found  by  clicking  on  the  Library  button  on  the 
internal  website’s  home  page.  This  would  bring  up  the  document  search  app  described 
above,  but  with  access  to  all  of  the  documents  and  other  material  released  to  all  of 
DT&E. 

7.  Other 

In  addition  to  the  above,  the  internal  website  could  be  used  to  keep  users  infonned 
and  obtain  needed  data  from  the  community. 

As  described  in  the  previous  chapter,  the  DT&E  Forum  website  was  developed  with 
the  idea  in  mind  that  it  could  be  used  to  keep  the  users  infonned  -  a  place  where  users 
could  go  to  find  the  latest  information.  In  addition,  the  DT&E  Forum  was  designed  to 
allow  users  to  provide  several  different  types  of  information  including  lessons  that  had 
just  been  learned  and  feedback  on  ways  to  improve  the  website.  These  two  facilities 
could  be  provided  in  the  proposed  internal  website  as  well. 

The  DT&E  Forum  website  uses  a  CAC  login  process  to  control  access  to  the 
information  it  contains.  Data  on  the  card  is  also  used  to  identify  users  who  have  been 
granted  administrative  privileges.  This  same  process  could  be  used  to  control  which 
users  are  allowed  access  to  which  of  the  applications.  Additional  security  techniques  can 
be  incorporated  as  necessary. 

D.  Summary  and  Conclusions 

Analysis  of  the  material  presented  in  this  chapter  leads  to  the  following  conclusions: 

•  The  DT&E  Forum  website  developed  as  part  of  this  task: 

-  Can  be  viewed  as  a  website  with  one  application 

-  Can  be  easily  modified  to  support  many  applications 

-  Can  support  many  simultaneous  users  using  different  applications 

-  Contains  features  useful  to  many  applications  such  as: 

o  Both  simple  and  complex  search  mechanisms  for  finding  documents, 
charts,  etc. 

o  Both  search  mechanisms  support  searches  involving  multiple  keywords 
and  multiple  key  phrases  as  well  as  combinations  of  both 
o  A  comprehensive  database  storage  facility 
o  A  feature  to  fully  index  documents  to  support  the  searches 
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•  Tasks  at  IDA  are  developing  a  significant  amount  of  infonnation  that  may  be  of 
interest  to  a  wide  range  of  people  working  in  DT&E. 

•  Much  of  the  information  IDA  is  developing  as  part  of  the  DT&E  tasking  could 
be  made  available  to  those  working  in  DT&E  via  a  modified  version  of  the 
DT&E  Forum  website  (the  proposed  internal  website)  developed  as  part  of  this 
task.  Material  developed  or  acquired  by  the  ODASD(DT&E)  could  be 
distributed  in  this  manner  as  well. 
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Appendix  A 

An  Example  of  a  Best  Practice 


This  appendix  provides  an  example  of  the  best  practice  documents  developed  in  this 
task.  The  general  outline  for  a  best  practice  is  shown  in  Figure  2-3. 

BP  11  -  015:  Establish  a  Test  and  Evaluation  Working-Level  Integrated  Product 
Team  (T&E  WIPT) 

A.  Best  Practice 

Charter  a  T&E  WIPT  to  advise  and  develop  the  T&E  program  for  TEMPs  and 
acquisition  strategies,  cost  and  schedule  planning  for  tradeoff  analyses,  insertions  into 
programmatic  and  contractual  documents,  oversee  progress  in  testing  execution,  oversee 
data  management  and  configuration  tracking,  inform  progress  and  assessment  reviews, 
and  advise  on  T&E  program  issues. 

As  early  in  an  acquisition  program,  preferably  at  the  onset  of  a  materiel  solution 
decision,  the  program  office  should  charter  a  T&E  WIPT,  to  be  led  by  be  the  Chief 
Developmental  Tester  in  the  program  office,  and  with,  as  a  minimum,  the  following 
representative  membership: 

•  The  designated  Government  Lead  DT&E  Organization  (external  to  and 
independent  of  the  program  office) 

•  The  designated  Operational  Test  Agency  (OTA) 

•  Oversight  organizations  (OSD  or  Service/Defense  Agency  Headquarters 
depending  on  whether  the  program  is  on  the  OSD  T&E  Oversight  List) 

•  Members  of  the  Integrated  Test  Team  (ITT) 

•  Organizations  that  issue  certifications  and  accreditations  based  on  test  data 

•  All  evaluating  and  reporting  organizations  for  the  program 

•  Organizations  that  generate  data  for  the  program 

•  Organizations  that  require  T&E  data  for  the  program 

•  Other  supporting  or  participating  test  organizations  (PTOs)  when  appropriate 

•  Logistics  and  training  organizations  when  appropriate 

•  The  system  contractor  when  selected. 

The  basic  roles  and  responsibilities  for  a  T&E  WIPT  should  be  as  follows: 

•  Assistance  and  advice  in  T&E  strategy  development  and  T&E  program 
planning  and  for  cost  and  schedule  development  (See  BP  1 1-008).  In  particular: 
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-  Analysis  of  Alternatives  (AoA)  to  ensure  cost  of  T&E  is  considered  and 
assumptions  are  realistic 

-  Acquisition  Strategy,  Technology  Development  Strategy,  and  the  Life- 
Cycle  Sustainment  Plan  (LCSP) 

-  Contract  preparation,  solicitation,  review,  and  awarding  [Request  for 
Proposal  (RFP),  Statement  of  Work  (SOW),  Contract  Data  Requirements 
List  (CDRL)] 

-  Evaluation  plan 

-  TEMPs  and  TEMP  updates  for  all  acquisition  milestones 

•  Plans  of  action  for  all  acquisition  milestones,  certifications,  interoperability  and 
information  assurance,  user  involvement,  Government  and  contractor 
participation,  and  other  major  events. 

•  Proposals  of  tailored  documentation  and  milestone  requirements. 

•  Review  and  provision  of  early  input  to  strategic  and  contractual  documents. 

•  Coordination  of  T&E  WIPT  activities  with  the  ITT  and  Overarching  Integrated 
Product  Team  (OIPT)  members. 

•  Deliberate  the  resolution  of  issues  in  a  timely  manner. 

•  Assumption  of  responsibility  to  obtain  principals’  concurrence  on  issues,  as 
well  as  with  applicable  documents  or  portions  of  documents. 

The  T&E  WIPT  members  should  meet  quarterly  as  needed  to  understand  and  build 
program  strategies,  to  resolve  issues,  and  to  produce  or  influence  specified  products  or 
actions.  The  T&E  WIPT  should  operate  in  accordance  with  best  practices  to  be  followed 
by  any  Integrated  Product  Team  (IPT)  as  shown  in  Table  A-l. 
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Table  A-1.  Integrated  Product  Teams  Best  Practices  Checklist 


Open  Discussions  with  No  Secrets 


Do 

Don’t 

0  Engage  all  members  in  the  IPT  process  by  soliciting 
inputs  and  applying  active  listening  skills. 

<S>  Personalize  organizational  position. 

0  Know  your  team  members’  preferred  methods  of 
communication,  and  thoroughly  understand  their 
organizational  roles  and  operating  environments. 

Isolate  people.  IPTs  are  only  effective  when  all 
team  members  are  participating. 

0  Trust  and  accept  each  person’s  expertise  and 
advice. 

Leave  issues  unaddressed.  Unaddressed 
issues  tend  to  resurface  at  higher  levels  and 
often  drive  major  rework. 

0  State  the  extent  of  your  authority/empowerment  and 
immediately  identify  issues  which  are  beyond 
established  limits. 

Forget  to  document  actions/decisions. 
Documentation  provides  all  team  members  an 
opportunity  to  clarify  issues  and  a  historical 
record  of  decisions. 

0  Establish  and  stick  to  the  agenda  for  the  meeting. 
Establish  operating  procedures  which  allow  any  team 
member  to  redirect  side  issues  to  other  forums. 

0  Take  the  necessary  time  to  prepare  for  the  meeting 
in  advance.  Conduct  research,  and  pre-meeting 
coordination  necessary  to  optimize  the  time  used  in  a 
group  session. 

0  State  your  organization’s  agenda  and  position. 

Openly  discuss,  resolve,  and  when  required  elevate 
issues. 

Dedicated/Committed  Proactive  Participation 


Do 

Don’t 

0  Commit  yourself  to  the  objectives  of  the  IPT. 

Bring  a  personal  agenda/negative  attitude  to 
the  IPT. 

0  Represent  your  functional  area  without  bias. 

Bring  additional  support  staff 

0  Actively  seek  and  receive  input  of  others. 

Skip  meetings 

0  Come  prepared. 

Issues  Raised  and  Resolved  Early 


Do 

Don’t 

0  Ensure  that  a  structure  in  in  place  to  identify  issues 
(e.g.,  dedicate  a  portion  of  each  meeting  to 
raising/discussing  issues). 

<S  Raise  issues  outside  the  IPT  process  (i.e., 
no  end  runs). 

0  Attempt  to  resolve  issues  within  the  IPT.  When 
issues  cannot  be  resolved,  provide  a  complete 
description  of  the  pros  and  cons  of  unresolved  issues 
to  decision  makers. 

0  Quickly  elevate  unresolved  issues  that  are  impeding 
program  progress. 

0  Ensure  necessary  functional  responsibilities  are 
represented. 
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Charter  Laun 

Charter 

ch  Initiation 

Launch 

0  Obtain  senior 
management 
agreement  on  charter 
objectives. 

0  Proceed  without  a 
written  charter  or 
establishing 
resources. 

0  Launch  the  IPT  as 
soon  as  possible 
following  charter 
sign-off. 

0  Allow  the  launch 

to  be  cumbersome 

and  unfocused. 

0  Ensure  adequate 
resources  are 
available  (money, 
time,  and  people). 

0  Make  the  charter 
too  complicated. 

0  Ensure  IPT 
agreement  and 
understanding  of  the 
charter 

0  Discourage  open 
member 
participation. 

0  Ensure  charter  goals, 
objectives,  and 
schedules  are 
realistic. 

0  Ensure  IPT  members 
are  trained  prior  to 
launch. 

Issues  Raised  and  Resolved  Early 

Do 

Don’t 

0  Develop  approaches  to  provide  feedback  to  team  members  and  0  Ignore  subpar  performance, 
their  home  organizations. 

0  Communicate  this  approach  to  the  team  and  consistently  apply. 

0  Recognize  contributions  of  team  members. 

B.  Rationale 

An  overarching  goal  of  the  T&E  WIPT  should  be  to  expedite  approval  of  and 
oversee  execution  of  TEMPs  so  as  not  to  delay  program  execution,  to  inform  design 
reviews,  and  to  oversee  T&E  activities.  This  can  be  accomplished  by  early  involvement 
in  acquisition  programs  by  DT&E  personnel  starting  with  the  Program  Manager 
chartering  a  T&E  WIPT  with  the  Chief  Developmental  Tester  to  set  schedules  and 
locations  for  meetings  and  shape  agendas.  The  charter  should  identify  the  background, 
purpose,  goals,  membership,  and  governance  of  the  T&E  WIPT. 

Public  Law  111-23,  “Weapon  System  Acquisition  Refonn  Act  of  2009,”  May  22, 
2009,  established  DASD(DT&E)  to  provide  assistance  to  program  offices  as  they  are 
developing  their  acquisition  strategies  prior  to  the  start  of  development,  and  then  oversee 
program  office  efforts  to  implement  the  strategies.  The  DASD(DT&E)  is  expected  to 
review  and  approve  acquisition  planning  documents  for  major  defense  acquisition 
programs,  as  well  as  monitor  program  execution  (GAO- 11-806,  “Actions  Needed  to 
Address  Systems  Engineering  and  Developmental  Testing  Challenges”).  DASD(DT&E) 
participation  can  be  accomplished  by  means  of  membership  in  the  T&E  WIPT. 

In  July  1999,  the  USD(AT&L)  reaffirmed  the  Department’s  commitment  to  the 
integrated  product  and  process  development  (IPPD)  concept  of  using  IPTs  throughout  the 
acquisition  process.  Those  IPPD  and  IPT  concepts  are  described  in  Rules  of  the  Road:  A 
Guide  for  Leading  Successful  Integrated  Product  Teams.  This  guide  is  designed  to  assist 
the  program  manager  and  supporting  acquisition  community  in  developing  and  executing 
high-performance  IPTs.  The  purpose  of  IPTs  is  to  facilitate  decision  making  by  making 
recommendations  based  on  timely  input  from  the  entire  team. 
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Test  Strategy  IPT 

The  purpose  of  the  IPT  is  to  assist  in  outlining  the  Test  and  Evaluation 
Master  Plan  (TEMP)  for  a  major  program.  The  objective  of  such  an  IPT  is 
to  reach  agreement  on  the  strategy  and  plan  by  identifying  and  resolving 
issues  early,  understanding  the  issues  and  the  rationale  for  the  approach, 
and,  finally,  documenting  a  quality  TEMP  that  is  acceptable  to  all 
organizational  levels  the  first  time. 

Example  given: 

A  T&E  WIPT  was  chartered  to  support  the  Biometrics  Automated  Toolset  (BAT), 
field  tested  in  Kosovo  and  then  sent  to  Iraq  with  a  Marine  Corps  unit  in  2003,  which, 
along  with  other  WIPTs  used  in  the  program,  were  attributed  for  the  successful  rapid 
response  of  this  program  and  the  eventual  adoption  of  the  capabilities  by  other 
government  agencies. 

C.  Sources 

•  “Shaping  the  Way  Ahead:  Anny  Biometrics  WIPT  Kickoff,”  Defense  AT&L, 
July-August  2011,  pp.  27-32,  by  Nicole  Daniel,  Kevin  Trissell,  and  Richard 
Hansen. 

•  Rules  of  the  Road:  A  Guide  for  Leading  Successful  Integrated  Product  Teams, 
Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  and 
Assistant  Secretary  of  Defense  for  Command,  Control,  Communications  & 
Intelligence  (C3I),  Rev.  1,  1999. 

•  Department  of  the  Anny  Pamphlet  73-1  (DA  PAM  73-1),  Test  and  Evaluation  in 
Support  of  Systems  Acquisition,  Headquarters  Department  of  the  Anny, 
Washington,  DC,  30  May  2003. 

•  Defense  Acquisition  Guidebook,  Defense  Acquisition  University,  July  29,  2011, 
(Pending  Update) 

•  BP  1 1-008  -  Developing  T&E  Cost  and  Schedule  Estimates 
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Appendix  B 

Documents  Developed  and  in  Process 


This  appendix  has  four  sections:  the  best  practices  developed  and  entered  into  the 
DT&E  Forum;  best  practices  that  were  in  progress  when  the  effort  was  concluded;  best 
practices  considered  for  development;  and  additional  papers. 

Best  practices  are  primarily  aimed  at  two  distinct  types  of  users:  (1)  those  working 
in  or  supporting  a  program  office  in  a  T&E  organization  that  is  developing  or  executing  a 
DT&E  program  and  (2)  testers  having  to  design  specific  test  events  (Note:  no  distinction 
is  made  herein  between  the  two  types  of  users,  the  number  designators,  and  order  of 
presentation).  Each  best  practice  is  accompanied  by  a  checklist  of  recommended  actions 
to  realize  the  best  practice  and  the  rationale  for  it  -  the  rationale  having  been  derived 
from  “Lessons  Learned”  from  a  variety  of  sources  as  described  in  the  main  text.  These 
sources  are  hyperlinked  to  the  best  practice. 

The  second  section  contains  additional  best  practices  specifically  drawn  from 
lessons  learned  from  reports  and  podcasts  from  the  SETP. 

The  third  section  includes  potential  best  practices  that  could  be  derived  from  2011 
Proceedings  Papers. 

The  fourth  section  contains  the  names  of  two  additional  papers  considered  to  be  of 
interest  for  future  work  in  this  area. 
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Best  Practices  and  Lessons  Learned 


Developed  and  Entered  into  the  DT&E  Forum 
BPll-001v2:  Aircraft  Weapons  System  Evaluations  of  Air-to-Air  Weapons 

Perform  fighter  air-to-air  weapons  DT&E  in  the  mission-oriented  context  of  the 
total  weapon  system  (aircraft-aircrew-  weapon-under-test,  and  supporting  off-board 
systems)  executing  the  engagement  kill-chain  in  realistic  operational  environments, 
including  threat  representative  electromagnetic  environments  and  targets.  (WSEP 
Lessons  Learned) 

BPll-002v2:  Use  M&S  for  T&E  Planning,  Prediction,  and  Analysis 

Use  modeling  and  simulation  (M&S)  to  plan  test  events;  pre-test  predictions;  data 
analysis,  and  to  supplement  testing  when  live  events  are  not  available  or  affordable.  Use 
M&S  to  gain  insight  into  technical  and  functional  capabilities  using  live,  virtual,  and 
constructive  simulation  in  different  environments  and  scenarios,  and  to  provide  surrogate 
data  for  filling  out  test  scenarios  with  threats,  targets,  and  other  participants.  However, 
use  M&S  for  which  verification,  validation,  and  accreditation  (VV&A)  has  been  tailored 
to  ensure  that  the  model  is  appropriate  for  the  specific  application  to  be  used. 

BP11-004:  Incorporate  Operationally  Trained  Testers/Users/Operators  into  DT&E 
Events 

Plan  and  design  DT&E  events  to  incorporate  test  team  members  operationally 
trained  in  relevant  threat  and  battlespace  environments,  and  in  tactics,  techniques,  and 
procedures  (TTPs)  as  required  to  measure  performance  in  a  mission-oriented  context  and 
interoperability  events. 

BP11-005:  DT&E  Tracking  and  Confirmation  of  the  Technology  Readiness  Level 
(TRL)  of  Critical  Technology  Elements  (CTEs) 

Use  government  DT&E  personnel  for  tracking,  assessing,  and  reporting  maturation 
progress  of  CTEs  throughout  the  development  and  acquisition  process.  Require 
government  DT&E  responsibilities  in  plans  for  conducting  DT&E  events  as  needed  to 
assess  CTE  maturity  associated  with  each  materiel  solution,  including  technology 
maturity,  integration  risk,  manufacturing  feasibility,  and,  where  necessary,  technology 
maturation  and  demonstration  needs. 

BP1 1-007:  Aircraft  Stores  Compatibility  Developmental  Test  and  Evaluation 
(DT&E) 

Characterize,  explain,  and  resolve  major  aerodynamic  and  mechanical  aircraft  stores 
inflight  and  separation  compatibility  issues  using  ground  test  facilities  and  modeling  and 
simulation  prior  to  flight  testing.  (Eglin  AFB  Lessons  Learned  Database) 
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BP1 1-008:  Effective  and  Reliable  T&E  Cost  Estimates 

Develop  realistic  costs  estimates  to  plan,  execute,  and  report  a  robust  DT&E 
program  that  will  provide  early  detection  and  correction  of  developmental  deficiencies. 
Incorporate  these  costs  into  the  acquisition  strategy,  analysis  of  alternatives  (AoA), 
evaluation  plans,  and  Test  and  Evaluation  Master  Plan  (TEMP)  resource  sections;  and 
ensure  consistency  with  other  budgetary  and  programming  documentation  such  as  T&E- 
1,  R-2,  R-3,  and  R-4. 

BP11-013:  Rapid  Response  DT&E  for  Incremental  Improvements  and  Retrofitting 

Plan  and  execute  a  streamlined  DT&E  program  for  deployed  systems  requiring 
rapid  response  to  incremental  improvements  and  retrofitting  that  focuses  on  minimum 
essential  safety,  capabilities,  and  limitations  to  meet  “operationally  useful”  or 
“operationally  acceptable”  criteria  as  may  have  been  established  by  Combatant 
Commanders. 

BP11-014:  Configuration  Traceability  Impacts  on  Reporting  Test 

Identify  and  record  the  exact  configuration  of  the  system  under  test  (SUT),  and  of 
the  test  environment  including  threats  and  targets,  for  each  test  event.  Address  impacts  of 
deviations  from  expected  final  configurations  and  test  environments  when  evaluating  and 
reporting  test  results,  system  maturity,  and  mission  capability  to  decision  makers. 

BP1 1-015:  Establish  a  High  Performing  T&E  WIPT 

Charter  a  T&E  WIPT  to  advise  and  develop  the  T&E  program  for  TEMPs  and 
acquisition  strategies,  cost  and  schedule  planning  for  tradeoff  analyses,  insertions  into 
programmatic  and  contractual  documents,  oversee  progress  in  testing  execution,  to 
oversee  data  management  and  configuration  tracking,  to  infonn  progress  and  assessment 
reviews,  and  to  advise  on  T&E  program  issues. 

BP11-017:  Testing  Dive  Recovery  Error  Budgets  for  Automatic  Ground  Collision 
and  Terrain  Avoidance  Systems 

Perfonn  a  detailed  dive  recovery  error  budget  analysis  during  flight  test  planning  of 
automatic  ground  collision  and  terrain  avoidance  systems  as  a  flight  test  safety  special 
interest  item.  (SETP  Lessons  Learned) 

BP11-018:  DT&E  of  Unmanned  Vehicle  C2  Systems 

Involve  experienced  or  trained  operators  in  the  planning  and  execution  of  DT&E 
(including  early  DT&E)  and  Integrated  T&E  to  address  the  command  and  control  (C  ) 
utility  of  unmanned  or  autonomous  vehicle  systems  in  development,  but  carefully 
constrain  their  role  to  ensure  test  objectives  are  achieved  without  bias  from  legacy 
training  and  operations. 
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BP1 1-019:  Ensure  DT&E  Strategy  to  Provide  Sufficient  Data  to  Inform  Major 
Acquisition  Decisions 

Develop  DT&E  program  strategy  that  will  provide  sufficient  data  to  inform  each 
acquisition  milestone,  design  review,  and  the  Assessment  of  Operational  Test  Readiness 
(AOTR)  on  progress  of  the  acquisition  program  to  achieving  three  key  knowledge  points: 
maturity  of  technology,  product  design,  and  manufacturing  processes. 

BP1 1-022:  Milestone  A  TEMP  Preparation  for  Prototype  Developmental  Test  and 
Evaluation 

Use  personnel  from  government  DT&E  organizations  experienced  in  applicable 
prototype  technologies  to  help  prepare  the  Milestone  A  Test  and  Evaluation  Master  Plan 
(TEMP). 

BP1 1-024:  Use  Early  Operational  Experiences  and  Assessments  to  Confirm  DT&E 
Program 

Review  Early  Operational  Assessments  (EOAs)  and  Operational  Assessment  (OA) 
reports,  results  from  Limited  Utility  Tests  (LUTs),  early  training,  and  operationally- 
trained  subject-matter  expert  assessments  for  potentially  unsatisfactory  operational  issues 
or  OTA  concerns  where  changing  or  refocusing  the  DT&E  program  could  lead  to  earlier 
fixes  that  would  preclude  unsuccessful  Initial  Operational  Test  and  Evaluation  (IOT&E) 
or  Live  Fire  Test  and  Evaluation  Program  (LFT&E)  events. 

BP1 1-027:  DT&E  Participation  in  AoAs 

Review  and  provide  DT&E  inputs  to  acquisition  program  Analysis  of  Alternatives 
(AoAs)  to:  1)  assess  testability  or  evaluateability  of  each  solution;  2)  use  as  a  source  of 
information  to  develop  evaluation  plans,  test  designs,  and  TEMPs;  and  3)  ensure  that 
alternatives  consider  any  requirements  for  investments  in  new  T&E  capabilities  and 
resources. 

BP1 1-032:  Encourage  Embedded  Instrumentation  in  Acquisition  Strategy  and 
T&E  Planning 

Encourage  embedded  instrumentation  for  on-board  data  sensing  and  collection, 
storage  and/or  real-time  transmission  during  development  of  an  Acquisition  Strategy 
and/or  Technology  Development  Strategy  and  analysis  of  alternatives  (AoA);  and  specify 
T&E  requirements  and  provisions  in  the  statement  of  work  and  request  for  proposal. 

BP11-035:  Establish  an  Adjudication  Process  for  Reliability,  Availability,  and 
Maintainability  (RAM)  Data 

Charter  and  establish  a  RAM  Integrated  Process  Team  (IPT)  or  other  working  team 
of  stakeholders  to  provide  integrated  planning  and  oversight  of  RAM  data  collection, 
analysis,  categorization,  and  verification  for  adjudicating  what  should  be  scored  as  RAM 
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failures  during  developmental  test  and  evaluation  (DT&E)  and  operational  test  and 
evaluation  (OT&E). 


BP11-036:  Use  an  Integrated  Test  Team  for  Test  Planning,  Test  Execution 
Oversight,  and  Test  Data  Management 

Establish  and  fund  a  government-led  Integrated  Test  Team  (ITT)  that  includes 
matrixed  representatives  from  the  Lead  DT&E  Organization,  Operational  Test  Agency 
(OTA),  prime  contractor,  and  other  stakeholder  organizations  that  require  system  test 
data  to  accomplish  their  respective  roles  in  assuring  a  program’s  success.  This 
organization  must  operate  through  the  Production  and  Deployment  Phase  of  the  program, 
and  be  empowered  to  perform  test  planning,  oversee  test  execution,  and  manage  all 
program  test  data. 

BP1 1-038:  Operational  Realism  in  DTE  for  Rapid  Acquisition 

Emphasize  operationally  electromagnetic  realism  in  DT&E  from  the  onset  of  rapid 
acquisition  programs. 

BP1 1-045:  Milestone  A  Test  and  Evaluation  Master  Plan  (TEMP)  Preparation  and 
Development 

Utilize  the  Lead  DT&E  Organization  to  help  prepare  and  develop  a  Milestone  A 
TEMP  using  the  assistance  and  resources  of  the  T&E  WIPT  to  sort  out  the  critical 
technology  elements  (CTEs)  to  be  demonstrated  during  the  Technology  Development 
(TD)  phase,  and  to  develop  the  DT&E  strategy  and  resources  required  to  demonstrate 
achievement  Technology  Readiness  Level  6  in  a  mission-oriented  context. 

BP  11-052:  Testing  in  Realistic  Electromagnetic  Environments 

Test  systems  in  a  mission-oriented  context  with  realistic  military  and  civilian 
electromagnetic  environments. 

BP1 1-058:  Design  and  Implement  a  RAM  Data  Collection  Process 

Implement  a  RAM  data  collection  process  from  program  onset  that  provides  early 
learning  and  knowledge  of  design  and  integration  issues  that  can  be  used  to  demonstrate 
reliability  growth,  that  can  contribute  to  an  assessment  of  suitability,  and  that  will  help 
project  probability  of  mission  success  in  deployed  systems.  Data  should  be  collected 
from  contractor  laboratory  tests  and  experiments  for  intended  end  items,  as  well  as  during 
all  contractor  and  government  developmental  testing,  live  fire  testing,  and  operational 
testing. 

BP1 1-060:  Establish  Description  of  “Mission-Oriented  Context”  Early 

Design  and  perform  DT&E  that  will  be  in  a  “mission-oriented  context”  from  the 
onset  of  a  development  and  acquisition  program  in  order  to  relate  technical  performance 
and  functional  measures  to  the  intended  operational  missions  and  environments. 
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Maintain  the  pedigree  of  the  mission-oriented  context  such  that  a  consistent  definition  of 
“operational  realism”  can  be  injected  into  DT&E  and  Integrated  T&E  throughout  the 
acquisition  program’s  lifecycle. 

BP12-001:  Integrated  Master  T&E  Schedule  and  Planning  Factors 

Construct  DT&E  program  schedules  that  account  for  realistic  planning  factors  for 
capabilities  and  limitations  of  supporting  resources,  certification  and  investment  lead- 
times,  needs  for  retesting  and  excursions,  dependencies  on  operational  and  other 
developmental  programs,  and  unknown  unknowns. 

BP12-002:  DT&E  of  C4I/System  of  Systems 

Use  networked  live,  virtual,  and  constructive  (LVC)  simulation  to  supplement 
DT&E  of  C4I  and  systems-of-systems  to  evaluate  data  exchange  within  defined 
environments,  to  validate  interoperability  as  an  input  to  the  Net  Ready  KPP,  to  certify 
information  assurance,  and  where  applicable,  to  evaluate  end-to-end  kill  chain 
operational  capability  in  a  joint  and  dense  threat  environment. 

BP  12  -003vl:  Aircraft  DT&E  Planning  Factors 

Construct  DT&E  program  schedules  that  account  for  realistic  planning  factors  for 
aircraft  development  to  account  for  capabilities  and  limitations  of  supporting  resources, 
certification  and  investment  lead-times,  needs  for  retesting  and  excursions,  dependencies 
on  operational  and  other  developmental  programs,  and  unknown  unknowns. 
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Best  Practices  In-Progress 

(Derived  from  Proceedings  of  the  Society  of  Experimental  Test  Pilots) 

BP  11  -  004vl:  Air-to-Air  Refueling  (AAR)  Qualification  Testing 

Perform  AAR  qualification  and  certification  testing  using  engineering  analysis, 
ground  testing,  and  flight  testing  for  human  factors  evaluations,  visibility  assessments 
and  lighting  evaluations,  proximity  testing,  mechanical  and  fuel  compatibility  testing, 
failure  modes  testing,  handling  quality  evaluations  for  both  the  tanker  and  receiver,  and 
envelope  expansion  based  on  the  preliminary  air-to-air  refueling  envelope. 

BP  12  -  005vl:  Weapons  Qualification  and  Certification  Testing 

Perfonn  weapon/stores  compatibility  qualification  and  certification  flight  testing  in 
the  context  of  testable  requirements/specifications,  operationally  relevant  envelopes,  and 
usefulness  of  Basic  Aircraft  Limit  (BAL)  as  indicator  to  approve  to  warfighter 
justification  for  flight  clearance  requests  for  envelope  expansion  BAL  data. 

BP  12  -  006vl:  Flutter  Testing  for  Weapons/Stores  Compatibility 

Require  open-air  flight  flutter  testing  for  weapon/stores  compatibility  qualification 
and  certification  to  resolve  uncertainties  involving  computational  analysis  and  wind 
tunnel  testing  to  provide  accurate,  useful,  and  safe  operational  envelopes  for  the 
warfighter. 

BP  11  -  007vl:  Aircrew  Workload  Analysis 

Perfonn  aircrew  workload  analysis  as  an  integral  element  of  the  systems  DT&E 
program;  focusing  on  aircrew  decision  process  of  executing  time  sensitive  kill  chains 
required  to  accomplish  a  mission  for  highly  integrated  systems  using  fused  information 
from  both  on-board  and  off-board  systems.  Include  a  comparative  workload  assessment 
of  the  “system  under  test”  with  the  systems  it  will  replace  or  complement  when 
applicable. 
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Potential  Best  Practices  which  could  be  derived 
from  2011  Proceedings  Papers 

Developmental  Test  of  JSOW  C-l  on  the  F/A-18E/F  Super  Hornet  —  Podcast 
Potential  Best  Practice  on  Human  Factors,  Safety,  and  Planning  Factors 

Authors:  Lt  Scott  Johnson,  (USN)  and  CDR  Andrew  McFarland,  (USN) 

•  The  JSOW  C-l  variant  is  the  first  Net  Enabled  Weapons  to  be  fielded  within 
DOD  (projected  IOC  2013).  Adds  a  Link-16  and  new  Moving  Maritime  Target 
(MMT)  algorithms  which  will  allow  for  a  standoff  Anti-Surface  Warfare  (ASuW) 
capability  against  ships  at  sea. 

The  Two  Most  Ominous  Words  in  Flight  Test:  “ Simple ”  and  “ Straightforward .  ”  A 
Look  at  INTEGRATING  MINES  on  the  F/A-18E/F Super  Hornet  -  Paper  and 
Podcast 

Potential  Best  Practice  on  Weapons  Integration,  Unknown-Unknowns,  and 
Planning  Factors 

Authors:  LCDR  Joshua  Filbey,  USN  and  Kami  Slagle 
Lessons  learned  include: 

•  Any  store,  despite  its  age  or  ability  to  be  carried  on  many  aircraft,  must  be  proven 
to  be  viable  for  use  on  each  individual  aircraft.  We’re  All  in  this  Tosether  - 
Lessons  Learned  from  C-l 7  All-Weather  Formation  Testing  -  Paper  and 
Podcast 

Potential  Best  Practice  on  Safety  Planning,  Planning  Factors,  Realistic 
Environments 

Authors:  LtCol  James  Hanley,  (USAF)  and  Maj  James  Quashnock,  (USAF) 

The  paper  discusses  how  the  test  team  incorporated  lessons  learned  from  three 
different  fonnation  systems  to  confidently  test,  evaluate  and  field  the  C-l 7  Formation 
Flight  System. 

Project  Dragon  Spear:  Weaponizing  the  MC-130w  -  Paper 

Potential  Best  Practice  on  Lean  Test  Team,  End-to-End  Evaluation,  and  Planning  Factors 
Author:  Maj  Andy  Jutte,  (USAF) 

•  The  Project  Dragon  Spear  team  tested  and  validated  integrated  ISR  and  precision 
strike  solutions  using  a  modular  and  palletized  Precision  Strike  Package  concept. 

•  To  facilitate  the  rapid  acquisition  concept  of  Project  Dragon  Spear,  the  46th  Test 
Wing  stood  up  the  Dragon  Spear  Combined  Test  Force  (CTF),  composed  of 
subject  matter  experts  from  three  test  squadrons  in  the  areas  of  precision  strike, 
C4ISR,  and  special  operations  aircraft  flight  test. 
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Additional  Papers  of  Interest  for  Future  Work 


F/A-18C/D  Litening  Pod  Loads  Envelope  Expansion  Lessons  Learned  -  Paper  and 
Podcast 

This  paper  addresses  risk  mitigation  measures,  how  these  problems  were  identified 
and  solved  throughout  the  flight  test  program  and  lessons  learned  from  this  test  program 
that  can  be  passed  on  to  other  test  organizations. 

E-2D  Advanced  Hawkeye  Carrier  Suitability  and  Structural  Demonstration  -  Paper 
and  Podcast 

This  presentation  provided  lessons  learned  through  the  test  planning  process, 
buildup,  execution,  and  reporting  of  carrier  suitability  structural  demonstration  testing 
with  specific  emphasis  on  risk  mitigation  strategies  for  this  high  risk  testing. 
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Appendix  C 

DT&E  Forum  User’s  Guide 


This  material  is  provided  under  separate  cover  so  that  it  can  more  easily  be 
distributed  to  website  users.  It  consists  of  two  parts:  DT&E  Forum  Quick  Start  User’s 
Guide  and  an  annex  to  it,  DT&E  Forum  Administrator’s  Guide,  which  describes  the 
DT&E  Forum’s  administration  functions. 
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Appendix  D 

September  19,  2011,  Version  of  Website 
Requirements 


On  September  19,  2011,  DT&E  provided  the  IDA  team  with  the  final  formal 
version  of  the  concept  for  the  DT&E  Forum  website  in  Microsoft  Power  Point  format. 
The  nine  screens  from  that  file  are  shown  in  this  appendix. 
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Appendix  E 

AT&L  Deployment  Timeline  for  DT&E 
Forum  Website 


This  appendix  contains  the  steps  that  AT&L  provided  in  its  timeline  for  the 
deployment  of  the  DT&E  Forum  website. 
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DT&E  Deployment  Timeline 


DT&E  =  DT&E  Development/  Management  Team 

AT&L  =  eBusiness  Operations,  Information  Assurance  or  Portfolio  Management 

EITSD  =  Enterprise  IT  Services  Directorate  Information  Assurance,  Executive  Change  Control  Board  (ECCB) 

Product  Owner  =  Government  customer  or  designated  representative 

Item 

Task  (in  logical  order) 

Due  Date 

Assignee 

1 

Complete  the  RFC  and  Business 
Needs  Analysis  (BNA) 
Documentation.  Return  to  ATL 

ASAP 

DT&E 

Documentation  needs  to  be 
completed  within  first  two  weeks. 
Samples  provided  to  DT&E  by 

James  Monroe. 

2 

Designate  IAO  to  be  POC  for 
eMASS  and  DIACAP  Process 

AT&L 

Complete  as  of  10/14/11  by  James 
Cumming 

3 

Complete  and  return  DITPR 
Questionnaire 

ASAP 

DT&E 

Questionnaire  emailedto  Shelly 
Wilson  by  James  Monroe  Due  90 

days  prior  to  final  deployment 

3a 

Obtain  Access  to  DITPR 
https://ditpr.dod. mil/dodcio/?l 
ndexlD=2 

ASAP 

DT&E 

Pending 

Due  60  days  prior  to  final 
deployment 

4 

Register  DT&E  in  gMA55 
(Required  workflow  interface 
for  DIACAP). 

AT&L 

Complete  10/14/11. 

AT&L  has  done  this  for  you 

5 

Coordinate  for /Attend  ECCB 
for  Business  Approval. 

ASAP 

AT&L,  DT&E 

Pending  completion  of  item  1 

6 

Address  all  IA  controls,  and 
upload  applicable  artifacts  into 

eMASS 

DT&E 

Controls  were  sent  10/14/11  by 
James  Cummings  for  review. 
Pending  Review 

7 

Create  DITPR  &  Snap-  IT  Entry 

AT&L 

Pending  completion  of  Item  3 

8 

Forward  C&A  package  to  EITSD 
for  approval. 

AT&L 

Pending  completion  of  item  6 

9 

Complete  Application 
Development/  Coding 

DT&E 

10 

Deliver  all  code,  database  and 
installation/configuration 
instructions  to  AT&L  engineers 

DT&E 

E-2 


Item 

Task  (in  logical  order) 

Due 

Assignee 

Status 

Date 

11 

Run  Ounce/  Fortify  (code 
vulnerability)  scans  on  source 
code  and  send  results  to  DT&E 
developers  for  review, 
justification  or  remediation. 

AT&L 

Dependent  on  Item  10 

12 

Justify  all  Ounce  vulnerabilities 
or  modify  application  code  to 
remediate.  Return  all 
documentation  and/  or 
changes  to  AT&L 

DT&E 

Dependent  on  receiving  item  11 

13 

Submit  Ounce/  Fortify 
Documentation  to  ATL  IA  for 
review  and  approval 

AT&L 

Dependent  on  receiving  item  12 

14 

Once  approved  Install  Final 
application  code /databases 

into  ATL  MAE  STAGING 
environment  only  if  no  code 
changes  are  required  as  as,  a 
result  of  vulnerability 
remediation  (If  changes  are 
required  to  application  re¬ 
submit  new  code  for  new 
Ounce/ Fortify  Scans 
repeating  items  11  and  12). 

AT&L 

Dependent  on  receiving  item  13 

15 

EITSD  conducts  scan  of 

STAGING  servers  with 
application  code  loaded 
(subject  to  EITSD  scheduling) 

EITSD 

Advance  coordination  10  days 
required. 

Required  for  ECCB  scheduling 

16 

l&E  conducts  User  Acceptance 
Testing  in  STAGING 

environment 

DT&E/  Product 
Owner 

Can  coordinate  with  AT&L  for 
testing  space.  If  UAT  locates 
significant  areas  for  rework-  ATL 
will  reevaluate  timeline  and  adjust 
to  accommodate  additional  ounce 
scans  and  schedule  new  testing 
dates. 

17 

User  Acceptance  Testing  (UAT) 
Form  is  signed  and  returned  to 
ATL  by  product  owner  (GOV) 

DT&E  Product 
Owner/ GOV 

18 

AT&L  IA  approves  Ounce/ 

Fortify  scan  justifications 

AT&L 
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Item 

Task  (in  logical  order) 

Due 

Assignee 

Status 

Date 

19 

Results  of  EITSD  STAGING 
server  scan  delivered  to  AT&L 

EITSD 

20 

AT&L  submits  requestto 
include  DT&E  approval  on 

ECCB  agenda 

AT&L 

21 

Authority  To  Operate  (ATO) 
issued  prior  to  deployment 

EITSD 

Items  19  and  20  may  be 
completed  in  any  order,  but  both 
must  be  approved  to  go  to  PROD 

22 

Attend  ECCB  and  obtain 
production  approval 

EITSD/AT&L 

Items  19  and  20  may  be 
completed  in  any  order,  but  both 
must  be  approved  to  go  to  PROD 

23 

Submit  System  Change 
Notification  (SCN)  for  approval 
with  ECCB  approval,  IA 
approval,  UAT  attached 

AT&L 

24 

Forward  copy  of  the  DT&E  ATO 
to  l&E 

AT&L 

25 

Deploy  DT&E  to  production 

servers 

AT&L 

Deployment  date  dependent  on 
completion  of  previous  tasks. 

Planning  Note:  Steps  9-  25  take  approximately  30-  40  days  to  fully  execute. 
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Appendix  F 

AT&L  Documents  Completed  for  DT&E 

Forum 


To  ensure  the  widest  distribution  of  this  document,  this  appendix,  which  is  FOR 
OFFICIAL  USE  ONLY,  is  published  under  separate  cover.  It  contains  the  six  documents 
that  AT&L  required  DASD(DT&E)  to  submit  as  part  of  the  effort  for  AT&L  to  host  the 
DT&E  Forum  website  on  one  of  AT&L’s  servers: 

•  e  Business  Needs  Analyses  (BNA) 

•  Memorandum  of  Agreement  (MO  A) 

•  Request  For  Change  (RFC) 

•  Application  Operational  Requirements  Questionnaire 

•  DoD  IT  Portfolio  Repository  (DITPR) 

•  Privacy  Impact  Assessment  Form  (DD  Fonn  2930) 
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Appendix  H 

Glossary  of  Terms  Used  in  Best  Practices 


ACAT 

Acquisition  Category 

ADM 

Acquisition  Decision  Memorandum 

AFOTEC 

Air  Force  Operational  Test  and  Evaluation  Center 

AHE 

Advanced  Hawkeye 

AIS 

Automated  Infonnation  System 

AO 

Action  Officer 

AoA 

Analysis  of  Alternatives 

AOTR 

Assessment  of  Operational  Test  Readiness 

APB 

Acquisition  Program  Baseline 

ART 

Available  Reaction  Time 

ASR 

Acquisition  Strategy  Report 

ATEC 

Anny  Test  and  Evaluation  Command 

ATIRCM 

Advanced  Infrared  Countermeasures 

ATS 

Automated  Test  Systems 

BAT 

Biometrics  Automated  Toolset 

BIC 

Business  Initiatives  Council 

BLRIP 

Beyond  Low  Rate  Initial  Production 

C2 

Command  and  Control 

CAPE 

Cost  Assessment  and  Program  Evaluation 

CBA 

Capabilities  Based  Assessment 

CDD 

Capability  Development  Document 

CDR 

Critical  Design  Review 

CDRL 

Contract  Data  Requirements  List 

CDT 

Chief  Developmental  Tester 

CFD 

Computational  Fluid  Dynamics 

CFIT 

Controlled  Flight  Into  Terrain 

CJCSI 

Chairman,  Joint  Chiefs  of  Staff  Instruction 

CJCSM 

Chairman,  Joint  Chiefs  of  Staff  Manual 

CND 

Computer  Network  Defense 

CNO 

Chief,  Naval  Operations 

COI 

Critical  Operational  Issue 

COMOPTEVFOR 

Commander,  Operational  Test  and  Evaluation  Force 

CONOPS 

Concept  of  Operations 

COTS 

Commercial  Off-The-Shelf 

CPD 

Capability  Production  Document 

CPI 

Critical  Program  Information 
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CTA 

Capstone  Threat  Assessment 

CTE 

Critical  technology  element 

CTF 

Combined  Test  Force 

CTP 

Critical  Technical  Parameter 

DAB 

Defense  Acquisition  Board 

DAES 

Defense  Acquisition  Executive  Summary 

DAG 

Defense  Acquisition  Guidebook 

DAMIR 

Defense  Acquisition  Management  Information 
Retrieval 

DAPS 

Defense  Acquisition  Program  Support 

DCACAS 

Data  Collection,  Analysis,  and  Corrective  Action 
System 

DCAPE 

Director  of  Cost  Assessment  and  Program  Evaluation 

DCR 

DOTMLPF  Change  Request 

DDR&E 

Director,  Defense  Research  and  Engineering 

DDT&E 

Director  of  Developmental  Test  and  Evaluation 

DIACAP 

DoD  Infonnation  Assurance  Certification  and 
Accreditation  Process 

DISA 

Defense  Infonnation  Systems  Agency 

DMS 

Digital  Modeling  and  Simulation  (Simulation  Facility) 

DODAF 

Department  of  Defense  Architecture  Framework 

DOE 

Design  of  Experiments 

DOT&E 

Director  of  Operational  Test  and  Evaluation 

DOTMLPF 

Doctrine,  Organization,  Training,  Materiel,  Leadership 
and  Education,  Personnel  &  Facilities 

DPS 

Defense  Planning  Scenarios 

DR 

Deficiency  Report 

DSB 

Defense  Science  Board 

DT&E 

Developmental  Test  and  Evaluation 

ECM 

Electronic  Counter  Measures 

EISS 

Enhanced  Integrated  Sensor  Suite 

EMD 

Engineering  and  Manufacturing  Development 

EMP 

Electro-Magnetic  Pulse 

EO 

Electro-Optical 

EOA 

Early  Operational  Assessment 

FCB 

Functional  Capabilities  Board 

FOC 

Full  Operational  Capability 

FOT&E 

Follow-On  Test  &  Evaluation 

FRP 

Full  Rate  Production 

GCAS 

Ground  Collision  Avoidance  System 

GCTA 

Ground  Collision  and  Terrain  Avoidance 

GLOC 

G-Induced  Loss  of  Consciousness 
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HIS 

HWIL 


Human  Systems  Integration 
Hardware-in-the-loop  (facility) 


IA 

Information  Assurance 

ICD 

Initial  Capability  Document 

ICE 

Independent  Cost  Estimate 

IFR 

Instrument  Flight  Rules 

IL 

Integration  Laboratory 

IOC 

Initial  Operational  Capability 

IOT&E 

Initial  Operational  Test  and  Evaluation 

IPL 

Integrated  Priority  List 

IPPD 

Integrated  Product  and  Process  Development 

IPT 

Integrated  Product  Team 

IR 

Infrared 

IRT 

Independent  Review  Team 

ISP 

Information  Support  Plan 

ISTF 

Installed-system  test  facility 

IT 

Information  Technology 

IT&E 

Integrated  Test  and  Evaluation 

ITT 

Integrated  Test  Team 

JCB 

Joint  Capabilities  Board 

JCIDS 

Joint  Capabilities  Integration  and  Development  System 

JITC 

Joint  Interoperability  Test  Command 

JMETC 

Joint  Mission  Environment  Test  Capability 

JRMET 

Joint  Reliability  and  Maintainability  Evaluation  Team 

JROC 

Joint  Requirements  Oversight  Council 

JTRS 

Joint  Tactical  Radio  System 

JUON 

Joint  Urgent  Operational  Need 

KM/DS 

Knowledge  Management/Decision  Support 

KPP 

Key  Performance  Parameter 

KSA 

Key  System  Attribute 

LCCE 

Life  Cycle  Cost  Estimate 

LCSP 

Life  Cycle  Sustainment  Plan 

LDTO 

Lead  DT&E  Organization 

LFT&E 

Live  Fire  Test  and  Evaluation 

LRIP 

Low  Rate  Initial  Production 

LVC 

Live,  Virtual,  or  Constructive 

M&S 

Modeling  and  Simulation 

MAC 

Mission  Assurance  Category 

MAIS 

Major  Automated  Infonnation  System 

MBTD 

Mission  Based  Test  Design 
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MCE 

Mission  Control  Element 

MCOTEA 

Marine  Corps  Operational  Test  and  Evaluation  Activity 

MDA 

Milestone  Decision  Authority 

MDAP 

Major  Defense  Acquisition  Program 

MDD 

Materiel  Development  Decision 

MF 

Measurement  facility  (precision  measurement  facility) 

MIDS 

Multifunctional  Information  Distribution  System 

MoE 

Measure  of  Effectiveness 

MoP 

Measure  of  Performance 

MoS 

Measure  of  Suitability 

MS 

Milestone 

MSA 

Materiel  Solution  Analysis 

NR  KPP 

Net-Ready  Key  Performance  Parameter 

NSS 

National  Security  System 

OA 

Operational  assessment 

OAR 

Open-Air  Range 

OCA 

Office  of  Cost  Assessment 

OIPT 

Overarching  Integrated  Product  Team 

OPLANS 

Operation  Plans 

ORD 

Operational  Requirements  Document 

OSD 

Office  of  the  Secretary  of  Defense 

OT&E 

Operational  Test  and  Evaluation 

OTA 

Operational  Test  Agency 

OTRR 

Operational  Test  Readiness  Review 

P&D 

Production  and  Deployment 

PARCA 

Performance  Assessment  and  Root  Cause  Analysis 

PDR 

Preliminary  Design  Review 

PEO 

Program  Executive  Officer 

PM 

Program  Manager 

PMO 

Program  Management  Office 

POR 

Program  of  Record 

PPP 

Program  Protection  Plan 

PSR 

Program  Support  Review 

PTO 

Participating  Test  Organization 

RAM 

Reliability,  availability,  and  maintainability 

RAM-C 

Reliability,  Availability,  Maintainability  -  Cost 

RFP 

Request  for  Proposal 

ROE 

Rules  of  Engagement 

RTO 

Responsible  Test  Organization 

SAE 

Service  Acquisition  Executive 

SEP 

System  Engineering  Plan 

SIL 

System  Integration  Laboratory 
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SME 

Subject  Matter  Expert 

SOW 

Statement  of  Work 

STA 

System  Threat  Assessment 

STAR 

System  Threat  Assessment  Report 

STAT 

Scientific  Test  and  Analysis  Techniques 

STR 

Software  Trouble  Report 

SUT 

System  Under  Test 

SVP 

Sound  Velocity  Profile 

T&E 

Test  and  Evaluation 

T&E  WIPT 

T&E  Working-level  Integrated  Product  Team 

TDS 

Technology  Development  Strategy 

TDSB 

Test  Data  Scoring  Board 

TEMP 

Test  and  Evaluation  Master  Plan 

TEMPEST 

Transient  Electro-Magnetic  Pulse  Emanation  Standard 

TES 

Test  and  Evaluation  Strategy 

TIR 

Test  Incident  Report 

TLAM 

Tomahawk  Land  Attack  Missile 

TRA 

Technology  Readiness  Assessment 

TRL 

Technology  Readiness  Level 

TRMC 

Test  Resource  Management  Center 

TSPI 

Time  Space  Position  Infonnation 

TTP 

Tactics,  Techniques  and  Procedures 

UAV 

Unmanned  Aerial  Vehicle 

use 

United  States  Code 

USD(AT&L) 

Under  Secretary  of  Defense  (Acquisition,  Technology 
&  Logistics) 

V&V 

Verification  and  validation 

VISTA 

Variable  Stability  In-Flight  Simulator  Test  Aircraft 

VV&A 

Verification,  validation,  and  accreditation 

WAS 

Wide  Area  Search 

WBS 

Work  Breakdown  Structure 

WIPT 

Working  Integrated  Product  Team 
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Appendix  I 

Acronyms  and  Abbreviations  in  this  Document 


AIAA 

American  Institute  of  Aeronautics  and  Astronautics 

AAR 

Air-to-Air  Refueling 

AoA 

Analysis  of  Alternatives 

AOTR 

Assessment  of  Operational  Test  Readiness 

BAL 

Basic  Aircraft  Limit 

BAT 

Biometrics  Automated  Toolset 

BNA 

Business  Needs  Analyses 

BPLL 

Best  Practices  and  Lessons  Learned 

C2 

command  and  control 

CAC 

Common  Access  Card 

CDT 

Chief  Developmental  Tester 

CDRL 

Contract  Data  Requirements  List 

CTE 

Critical  Technology  Element 

DASD(DT&E) 

Deputy  Assistant  Secretary  of  Defense,  Developmental 
Test  and  Evaluation 

DAU 

Defense  Acquisition  University 

DITPR 

DoD  IT  Portfolio  Repository 

DSB 

Defense  Science  Board 

DT&E 

developmental  test  and  evaluation 

DTAP 

Defense  Technology  Area  Plan 

EITSD 

Enterprise  IT  Services  Directorate 

EOA 

Early  Operational  Assessment 

GAO 

Government  Accountability  Office 

I&E 

Installations  and  Environment 

IDSS 

interoperability  decision  support  system 

IPPD 

integrated  product  and  process  development 

IPT 

Integrated  Product  Team 

ITEA 

International  Test  and  Evaluation  Association 

ITT 

Integrated  Test  Team 

LCSP 

Life-Cycle  Sustainment  Plan 

LDTO 

Lead  Developmental  Test  and  Evaluation  Organization 
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LFT&E 

Live  Fire  Test  and  Evaluation  Program 

LUT 

Limited  Utility  Test 

LVC 

live,  virtual,  and  constructive 

M&S 

modeling  and  simulation 

MOA 

Memorandum  of  Agreement 

MUOS 

Mobile  User  Objective  System 

OA 

Operational  Assessment 

ODASD(DT&E) 

Office  of  the  Deputy  Assistant  Secretary  of  Defense  for 
DT&E 

OIPT 

Overarching  Integrated  Product  Team 

OT&E 

operational  test  and  evaluation 

OUSD(AT&L) 

Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics 

POA&M 

Plan  of  Action  and  Milestones 

POC 

point  of  contact 

PTO 

participating  test  organization 

RAM 

Reliability,  Availability,  and  Maintainability 

RFC 

Request  For  Change 

RFP 

Request  for  Proposal 

SETP 

Society  of  Experimental  Test  Pilots 

SOW 

Statement  of  Work 

SUT 

system  under  test 

T&E 

test  and  evaluation 

T&E  WIPT 

T&E  Working  Level  Integrated  Product  Team 

TD 

Technology  Development 

TEMP 

Test  and  Evaluation  Master  Plan 

TRMC 

Test  Resource  Management  Center 

TTPs 

tactics,  techniques,  and  procedures 

UAT 

User  Acceptance  Testing 

UI 

user  interface 

VV&A 

verification,  validation,  and  accreditation 

WIPT 

Working-Level  Integrated  Product  Team 

WSEP 

Weapons  System  Evaluation  Program 
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